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TEXT STORAGE AND RETRIEVAL SYSTEM 
AND METHOD 

FIELD OF THE INVENTION 

This invention is a method and system for storing and 
retrieving infonnation. More particularly, it is a method and 
system compressing, storing, searching, retrieving, and 
decompressing informatioa written as text. 

BACKGROUND OF THE INVENTION 

Text is the primaiy way that human knowledge is stored, 
and, after speech, the primary way it is transmitted- As it is 
commonly rcjHcsented in computer systems, however, text 
is a particularly inefficient and unmanageable form of infor- 
mation. Its usual format— a linear series of bytes encoding 
characters, blank spaces, punctuation symbols, and format- 
ting codes — is ideal for generating and editing documents, 
such as in a word processor The linear format makes it ^ 
extremely easy to insert, ddcte, and replace individual 
characters in a document Unfortunately, this same format is 
also most commonly used to store text documents, long aflcr 
the text is written and needs no further revisicm. 

As a storage format, a linear sequence of bytes rqxesent- 25 
ing characters canies significant penalties, in tx>th storage 
size and access speed. Typically, each byte in a text docu- 
ment encodes a single letter or a single punctuation symbol. 
Altogether there are roughly ninety-four such characters In 
typical English text: the fifty-two letters (uppercase and 30 
lowercase), ten digits, and thirty-two common punctuation 
symbols, as represented in the American Standard Code for 
Information Interchange (ASCII). Because a single byte can 
represent one of 256 different values, &e remaining 162 
values are typically used Ijy English word processing pro- 35 
grams to encode formatting symbols, international 
characters, and ^)ecial punctuation marks. Some of diese 
codes are also used as flags, to indicate that later bytes in the 
linear sequence do not represent characters, but instead 
define structures such as columns, tables, or figures in the 40 
text, or indicate font changes. Since the great majority of 
bytes in a text document represent characters, however, each 
byte ''wastes" approximately two-thirds of its range. 

The linear storage fozmat similarly limits access to the 
underlying information in a text document. Text inf ormatioo 45 
is embodied in words, not characters, and furthermore 
relatively few of these words are significant The only way 
to locate an important, concept-defining word ("keyword") 
in a document stored as linear text is to scan the text using 
a pattern-matching algorithm Although advances in this so 
field — notably the Knuth-Mofris-Pratt and Boyer-Moore, 
text searching algorithms— can inqrove the speed of a linear 
word search over the traditional "naive" technique 
(sequentially comparing each byte in the document with 
each byte in the searched-fw word), a pattern-matching 55 
algorithm must still traverse the entire document to be sure 
that it has found every instance of a particular keyword. 
Moreover, in English, for example, nearly one half of the 
words in a doamient are articles, conjunctions, prepositions, 
or common modifiers, all of which have a very low inf<»- 60 
mation content Thus, a linear keyword search algorithm 
spends roughly half its time considering, and discarding, 
useless words. 

Attempts to ioo^ove the storage efficiency of text docu- 
ments have generally been made along lines entirely scpa- 65 
rate from atten^)ts to improve access efficiency. This is 
because increasing storage efficiency ("conqn-essing" a 
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document) tends to decrease access speed, since tiie oom- 
picssed document must be decompressed before it is 
searched. Similariy, increasing access efficiency ("indexing" 
a document by building a list of keywords in the document) 
3 is usually accompanied by a decrease in storage efficiency, 
because indexes and related structures, such as pointer tables 
or signature files, require additional storage space. 

Tbxt compression techniques have generally focused on a 
linear replacement of bytes in the document with codes that 
either (1) individually rqxresent more than one byte in the 
original document, or (2) individually require fewer bits (on 
average) than an eight-bit byte to represent the same char- 
acter. The L/cmpd'Tiy algorithm, and its variants, is an 
exanq>le of the first type of compression; the static Huffinan 
code is an example of the second. Both techniques achieve 
a conqression ratio of between 50% and 70% for English 
text (resulting in a con^iressed document that is between 
30% and 50% its original size). Each of these techniques, 
however, again (voduces a linear document which must 
furthermore be decompressed, at least in part, before it is 
searched. Moreover, both techniques require expensive (in 
terms of speed) bit naanipulations that can be difficult to 
implement on machines not rich in partial word or partial 
byte instructions. 

Text indexing techniques, on the other hand, have gener- 
ally focused on building an external index, signature file, or 
similar structure, that encodes, for quick retrieval the impor- 
tant keywortls in a set of documents. These systems either 
rely on a human operator to identify the documents* 
keywords, or, as in most automated systems, simply index 
every word in the document set, excqAing very common 
words (called "stopwOTds"). The keywords for indexed 
documents can be searched extremely quickly: the search 
usually consists of simply reading die index to discover 
which documents contain a particular keyword. 
Unfortunately, the index or signature files often require 
considerable storage space. A cooqjrdiensive index, for 
example, whidi contains document and word position infor- 
mation for every significant word in a document set, is 
usually from 50% to 100% the size of the original document 
set 

There is thus a need for a m^bod and system that 
increases both storage efficiency and access efficiency fOT 
text documents. This invention achieves both of these goals. 

SUMMARY OF THE INVENTION 

This invention is a system and method for compressing, 
storing, searching, retrieving, and decompressing informa- 
tion written as text The system and method provide the 
significant advantage of simultaneously ijxq)roving both the 
storage efficiency and the access efficiency for text docu- 
ments. The system and method create a highly comj^esscd 
data structure that allows the text of documents stored in the 
structure to be keyword searched and, furthermore, proxim- 
ity searched, in their native, confessed format. 

The system and method of this invention conqmse a 
lexical parser which divides the text of a document into 
search terms (Le„ those words of the document that have 
medium to hi^ information content) and all other informa- 
tion. "All other information" includes the punctuation 
between the search terms, "stopwoxds" (those words that 
have low information content such as the and of), and 
formatting information. The system and method of this 
invention further store the search tenns solely in an inverted 
structure; that is, the search terms are stored uniquely, 
together with the list of all positions in the docuoKnt at 
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which those terms occur. The system ajKl method of this This cooqxession technique, termed block compression in 
invention also store the "other information'* of the document this ^dfication, uses a uniquely generated list of sub- 
in an auxiliaiy data structure. The inverted structure. strings to compress strings — such as the search terms and 
together with the auxiliary data structure* contain sufiBdent the "other information** lying between search terms identi- 
inf(»mation substantially to reproduce the original text of 5 fied by the lexical parser of this invention — which com- 
thc document The inverted structure which stores the search mooly contain tliose substnngs. Block compression provides 
terms provides the significant advantage of allowing die the significant and unexpected advantages of (1) compress- 
document to be easily kcywcsd searched and proximity ing infonnation; (2) reducing the "unit count** (i.e., the 
searched, because keywords (i.e., the search terms) and their number symbols in the compressed string) of the com- 
positions within the document may be quickly looked up in pressed information; (3) providing increased con^resslon 
the inverted structure, in rmich the same way one uses the for longer strings, at the expense of providing decreased 
index of a book. Moreover, because the search terms of the compression for shorter strings — an advantage for fixed- 
text are stored solely in the inverted structure, the system length storage structures; and (4) providing increased com- 
and method of this invention provide a significant degree of pression for low information content strings (such as wonis 
compression in staring the document formed of long, but common prefixes or sufiSxes, like 
In accordance with another objective of this invention, die anthropology or determination), at the expense of decreased 
positions corresponding to tt>e search terms stored within die compression for high information content string? (such as 
inverted structure are assigned in a maimer tiiat counts each nonsense words, codes, or abbreviations, like int'l or xr4ti) 
search term within the document as oocq>ying a single -an advantage for keyword storage and searching systems, 
position, and each instance of "other information** in die where the entire length of a comnkon keywofd must be 
document as one or more positions. This aspect of die 30 stored, to distinguish it fi^om other words, but where only die 
invention provides die signfficant advantage of assigning first few characters of a nonsense word need be stored to 
positions to die search terms of die text such duU die distinguish it from others. F6r diese reasons, block com- 
positions can be used bodi fOT proximity searches and for pression is particularly useful in compressing die search 
reconrtwcting the text of the dooiuncnt These p<witions, ^ms and "odicr information** in the preferred embodiment 
termed fiizzy positions m dus specification, stnkc a balance 25 of diis invention 

between providing reasonably accurate infonnation regard- , , * ^ ^ - 

ing die relative positions of search tmns within die text! and ..^"SST^'I "1^?^^*^ 'Z'!^''^^ 

providing information regarding die position and size of die « smgutoization me^od for words is provi^ The 

^cr Samation** I3S betleen thT search terms The ' ^vcrsibte dngulan^on mcdiod is c^^ 

dualutilityofthese"fu^**S^S^^S^ 30 Tlt'fi:^ftTJ ^-Jf T"^ 

storage efficiency and inacaTed accTss efficiency for text ^ '^f P^^^i*^^ ^F^' 

docmMsnts "'"waw cmwcucy i« u^u advantage of ensuring tiiat die second form is always 

' ^ . capable of being transformed back into die OTiginal (plural) 

In accordance widi anodier objective of dus invenUon, die ^vord, widiout crrw. This advantage is particularly useful for 

search terms of die document are converted to dicir lemmas, performing morphological transformations in die prefored 

or root forms, before stmge in the inverted structure. This 35 embodiment of this invention, because it is cq>able of 

aspect of the invention increases the storage efficiency of the converting plural or plural-£^>pcaring jargon or proper 

structure by requiring the storage of only a single fonn of nouns, such as metacodes or Addickes, into an ^Tproximate 

related search terms, and simultaneously improves access singular form diat can always be converted back into die 

efficiency for keyword searching because only a single form csiginal word. 

of a keyword need be looked up hi die inverted structure to 40 ^RIEF DESCRIPTION OF THE DRAWINGS 

retrieve instances of all forms of die keyword in die docu- „^ , . ..... 

iQjgiit, FIG. 1 is a block diagram showing the relationship 

die search terms of the document are stored m the inverted '»--wii,-i- w a* 

stnirtute by i«feceace to a second «aili«y data structure 45 l^TLtTZ^^JL^^°^ 

whidi con^s those tenns. This a^ertSf the invention * !"«^* "^J^Tf^ . , 

provides .he ^gnillcan. advant^ jrK.ving the st<.age L^SSoff^:!!'' "'^^ 

cffiaency of the search terms in the inverted structure, by err* a ui^w c i j * 

requiring .hen. ,0 be stored only by refaence; and simuJ 1^^^^^ 

neously mmroves access efficiency, where multiple docu- vi ct/- ui u u . ^ * 

meats are Xed. by placing (he aerial scaidi t«ii of aU of ^^J^ * bloclcdu.gnun showmg the substnictures of an 

die documents in one location, where they may be easily ^ i t^i t a- u • ^ r 

'^J^Tt ' ^y^'.-^ ^.^ a^^^^'t^Se**^ 

data stmcture of the preferred emboduneiit composes more TTrr^u ui uj- ^ - ^ ^ r i_. 

dian one file^ diagram showing die procedure for block 

^ compressing a search term, 

m acD^ce y^^«^<^^J?«ive of dusinyention^ FIG. 8 is a block diagram showing the procedure for 

Ac search tarns (whether stored directly in die uivertcd generating block tables to die block wmpressors used in 

structure, or stOTcd by reference to an auxiliary structure) arc die preferred embodiment 

compressed. Similyly, the Ust of positions stored in die FIG. 9 is a block diagram showmg the procedure for 

mvCTtcdstructurcofthismventionmayaisobccomiTCSsed. 60 performing a pure kcyword-in-document search of the 

Both of these aspects enhance die storage efficiency of die archive. 

^stem, whfle not significantiy affecting access efficiency, piG. 10 is a block diagram showing die procedure for 

because only a portion of die document (i.e., die pcxtion extracting a metafile from die archive, 

containing die search terms) need be decompressed to FIG. U is a Wock diagram illustrating die transformation 

conduct a keyword or proximity search. 63 of a metafile's word vector table into a position list in 

In accordance widi anodier objective of diis invention, a preparation for expanding a metafile into die original 

highly advantageous conqxession technique is provided document 
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DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

L Overview 

The system and method of this invention transfonn one or 
more text documents into a data strucdine termed an archive 
of the documents. The archive structure Is highly 
corniHessed, requiring far iess storage space than the original 
documents, but it contains all of the information necessary 
to rebuild the original documents. In this sense, the inven- 
tion is similar to multiple-document compressing or 
archiving systems such as the UNIX *tai^ and **zip** 
programs, and the IBM-PC *))kzip** and ^Iharc** utilities. 
However, the archive stnicture of this iDveotio& achieves 
better compression than these programs. paiUy because the 
structure is designed to com|nvss only text. 

More inq>ortantly, however, the archive structure stores 
the search terms of all of the original documents (excepting 
a handful of conmKm words, called stopwords) in a compact 
'Inverted** structure. (An inverted file or data structure is one 
in which each unique term in the document is stored, 
followed by a list of its positions within the document In a 
linear file, by contrast, terms are stored sequentially in tiie 
order in which they appear.) Because the search terms of die 
document are stored in an inverted structure — and only in an 
inverted structure — keyword searches of documents stored 
in the archive are exceptionally fast. In effect, documents in 
the archive are keyword-searchable in their native, com- 
pressed format: only a very small degree of deconq>ression 
is required, and oiUy for the segment of the archive that 
contains the search terms. 

FIG. 1 diagrams the basic stnicture of the preferred 
embodiment of this invention, and introduces some of the 
terms used in the discussion below, A text document 10 is 
parsed into a set of conqwnent structures, tenned a metafile 
20. The metafile 20 may then be added to the archive 50. The 
archive 30 can be searched to locate documents that match 
a keyword queiy 40; matching documents are first extracted 
from the archive into the metafile 20 format, and then 
expanded into the original document 10, for printing or 
editing in a word processor. 

IL The Metafile 

Although it is possible to convert a text document into its 
archive representation and add it directly to the archive 
structure, it is preferable to parse the document into an 
interim structure before adding it to the archive. This interim 
structure, termed a metafile, enables the user (or another 
system) to review the document, in the fonn in which it will 
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is added to the archive. If an error were to occur during the 
creation of the metafile, the archiving process could be 
safely halted, leaving the archive structure unchanged. 

As shown in FIG. 2. the m^afile 20 is a collection of 

5 separate conqx>nents that con^letely describe the original 
document 10. In particular, the metafile 20 contains sub- 
structures that individually describe (1) the unique words in 
the document (i.e.. each word in the document is listed once) 
together with a list of each w<xd's position(s) within the 

10 document — this is the metafile's word vector table 21; (2) 
the punctuation that appears between words in the 
document, as well as formatting codes (e.g.. typeface 
changes, new paragraphs) and special codes necessary to 
reconstruct accurately the document's text (e.g., recapital- 

. j ization flags, morphological transformation flags, and non- 
searchaUe terms)— this is the metafile*s code list 22; (3) the 
format of each paragraph in the document — this is the 
metafile* s paragraph descriptor table (POT) 23 and para- 
gr^b descriptOT reference list 24; and (4) miscellaneous 
elements of the document that cannot be efficiently stored in 

^ any of the other substructures — this is the metafile *s residu- 
als list 25. 

The metaliie*s word vector table 21 stores a list of aU of 
the unique w(xds in the document, together with a list of 
each word's positions(s) within the document As will be 

^ discussed below, however, the words stored in the word 
vector table 21 do not exactly correspond to the words in the 
document: instead, regular noun and verb inflections (in 
Engli^, for this embodiment) arc converted to their lemmas 
(a lemma is the uninflected base form of a w<»d); the w<Hds 

30 are all stored in lowercase only; and some common words 
(stopwords) are not stored here at all The positions associ- 
ated with each word in the word vector table 21 concspond 
roughly — but not exactly — to the location of the w<xd in a 
sequential list of all of the document's w<xds. That is, the 

35 first word of a document is assigned a position near 1, the 
tenth word is assigned a position near 10. the hundredth 
w<H^d is assigned position near 100. and so on. The assign- 
ment of positions to words is made by a lexical parser 120 
(described below), and. as mentioned, it is not exact; instead. 

^ it strikes a balance between the need to accurately recon- 
struct the docuizieDt. and the need to represent word posi- 
tions relative to one another with sufficient accuracy that 
proximity scardics (described below) can be reasonably 
accomplished. The positions of words stored in the word 
vector table 21 are therefore termed fuzzy positions. 

Table 1. below, shows the construction of an example 
word vector table 21 from the first paragrqA of J. R. R. 
Tolkdn's *The Hobbit.** 



TABUS I 



Ongmal documesl! 

Ao Uoezpected Paity 

la a bold in the ground (faere IhrnS a hobbiL Not a nacty, diny, wet bote, fiUed wiHi the 
axis of wonns md an oozy xor yet a dry» bore, nady bok wiifa nothing in it to iit 
dawn OP or to cat; it wm a hobbil-hc^ and that tncana ownfoat. 

^ferd vector table: 

bare 39; comfon 60; diity JD-, dorwn 48; <ky 37; eat 51; cod 27; fiO 23; ground 10; bobbit 
15. 54; t»fe 8, 23, 42, 56; live 13; mean 59; nasty 18; nodi 44; oozy. 31; party 5; sandy 
41; ait 47; amell 32; uncxpecl 3; wet 22; wonn 29; yet 35 



be Stored in the archive, before it is actually added to the 
archive. This may be necessary because some changes may 
be made to the document in the interests of storage 
efficiency, befcHC it is added to the archive. In addition, 65 
creating an interim structure allows nearly all of the parsing 
and compressing steps to be convicted before the document 



Unexpected, the first non-stopwccd in the document, is 
converted to lowercase, morphologically (ransfonned to its 
lenmia, unexpect, and then assigned a position of 3. 
Comfort, the last word in the document, is assigned a 
position of 60. In some cases, such as in the phrase **sandy 
hole,** the assigned word positions are truly adjacent: sandy 
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is given position 41, and one of hole's positions is 42. In 
other cases, however, qsparcntly adjacent words are sepa- 
rated by one or more positions, wtxich coirespond to codes 
required to reconstruct the original words and punctuation of 
the document: In **dry, bare,** dry and bare are assigned 
positions 37 and 39, respectively, because the intervening 
conmsa and space are given position 38; in "giound there 
lived** ground and live arc given positions 10 and 13, 
respectively, because the intcn^emng stopword there is 
assigned position 11, and an invisible re-stemming flag, 
which converts live to lived, is assigned position 12. Note, 
however, that a con^vession technique (described below in 
Section VI) applied to these intervening codes in the pre- 
ferred embodiment wilL in most cases, reduce the '*unit 
count** of the codes separating words, improving both stor- 
age efBcicncy and ttie accuracy of the word-position assign- 
ments. Thus, for example, in the phrase "bole, filled,** hole 
and fill arc given positions 23 and 25, respectively: the 
intervening comma and space and the re-stemming Sag that 
converts fill to filled are rq)resented by a single symbol at 
position 24. 

The codes which represent the punctuation, foffmatting, 
morphological transformations, and stopwords which occur 
between terms stored in the word vector table 21 are termed 
metacodes, because, like the metafile, they exist only 
tenqxH^rily, during creation of the archive structure. These 
codes are assembled in the sequence in which they occur in 
the original document 10, then ccxnpressed and stored in the 
m£tafile*5 code list 22. The set of metacodes used in the 
preferred embodiment is defined as shown in Table 2, below. 
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(underlines, italics, boldface) flags, and word completion 
codes fOT very long words not easily stand by the system. 
These codes are described nacre fully below. 

J nL Parsing a Text Document into a Metafile 

FIG. 2 illustrates the procedure f<x parsing a text docu- 
ment 10 into its metafile 20 representation. The odginal 
document 10 is first tested, by a type identifier function 100, 
to determine what type of document it is: ASCH text, for 

10 example, or a document created by a word processor sucfa as 
Word Perfect™. Documents created by programs such as 
word processors are commonly marked io some fashion to 
identify their creator. Word Perfea™ documents, for 
example, carry a short header that identifies the document as 

13 a Word Perfect™ document, and indicates what version of 
Word Perfect™ was used to create it Preferably, the type 
identifier function 100 tests fCH^ a series of anticipated 
document types, and only returns a document type of ASOI 
if no other type is discovered first 

^ The document 10 is then Unked to a docimient scanner 
110 which is suited to read documents of the type identified 
by the type Identifier function 100. The document scanner 
110 provides the basic service of returning, one item at a 
time, the characters and formatting codes of the document 

^ 10. The scanner 110 may also provide some translation 
services, such as converting extended characters deliberately 
not recognized by the rest ci the system (c.g., Greek letters, 
mathematical symbols) to approximately equivalent recog- 
nized characters. To offer the siiiq>lest example, a scaimer 
110 for an ASCn document might simply return, in 



TABLE 2 



1 . A set of ffjfnnxHT punctmtiao symbob. Id the {xefionvd ttabortimeot, tbe 
ftxUowmg symbols m iuiipcrtod: I ««|%&*0-M];:'" -'To- 
ll. Itine- and four-period ellipaet ue alao supported. 

2. Tbe set of al upwurJ a. Eacti slopwoid is isaigDed a aeparate metacocfe. . 

3. Tt» act of motpfaological mmftformatioo cods*. In the prefiored embodiment, 
the foUowiog codes are supported: caps, alfcapa. ^Mctalcaps, ptunl, 
pooaessive, pluflEd, phulog, eeskL (Tbe of tfaeae codes ate djanissed 
below.) 

4. Space— Che tpaoa cbsracter. 

5. Newpage— aignala the beginaipg of a new page In the docisnenl. 

6. Newpaxvgrapb— atgnals die beginning of a new paragraph. 

7. Focmat oo/off—CQggka a word emphasb style; is die preferred embodimeQl. 
leoognized wood emf^ta styba are mderline, bold&ce, and italics. 

8. Note — sigzmls the begnming or ending of a footnote or MKkiote. 

9. Repeat — mH«*af^ that the next meCttrodt is to be repeated a times (where n is 
prefeiably stored in the loaiduals list 25, as described below). 



To improve the storage efiSdency of die archive structure, 
the paragraph fonosx of the original document 10 is stored 
as a table and a list of references to that table, rather than 
embedding paragr^>h format information directly within the 
code list 22. For this reason, the list of metacodes contains 
only one paragraph formatting code, the new paragr£9>h 
signal The actual fonmats of paragraphs are stared in the 
metafile's 20 paragr^h descriptGr table (FDT) 23. (The 
entries in the PET 23, called paragraph deserter codes will 
be explained in more detail below.) The metafile 20 also 
contains a paragraph descr^nor reference list 24, which 
indicates, for each paragnq)h in the document 10 seriatim^ 
die paragrai^ descriptor code that is associated with that 
paragraph. 

Finally, any other infonnatioo about the docuntent 10 not 
easily or efEdently stored in the other four sutKtnictures of 
the metafile 20 is stxxed in the metafile*s residuals list 25. In 
the preferred embodiment, (he residuals list 25 contains 
miscellaneous information such as word emphasis 



„ sequence, each character in the document (converting 
unsupported characters, such as the " symbol, to a generic 
symbol such as @, along the way). Of course, in the 
preferred embodiment a slightly more com]^cx ASCII docu- 
ment scanner 110 is desirable, to account for example, for 
the diflferent end^f-line end-of-paragr^h delimiters 
commonly present in ASCD documents. (Preferably, this 
scanner 110 first determines which type of delimiter the 
document uses, and then converts each instance of that 
delimiter to the generic return symbol recognized by the rest 
of the system; see Tahlt 3, below). A scanner 110 for Word 

60 Perfect™ documents, by contrast, must not only return die 
characters of die document 10, but must also convert Word 
Perfect™ formatting codes to equivalent formatting codes 
recognized by die rest of the system. 
By using a document scanner 110 keyed to die original 

6S document*s type, a wide variety of documents 10 can be 
converted directly into metafile 20 representations, wittiout 
the need for intermediate file conversions. To accomplish 
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this goal, the system's fonnat analyzer 130 and lexical 
paiser 120 (both described in more detail below) are each 
designed to accept the sequence of single characters or codes 
returned by the scanner 110. In the preferred embodinient, 
the scanner 110 retunii either the chafacters of the document 
10, or one of the generic formatting codes listed in Table 3, 
below. 
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boundaries of the current line, in the line accumulator 132, 
and the boundaries, leading, and format of the current 
paragraph, in the paragraph accumulator 133. 

If the document 10 is line-delimited, then the accumulator 
engine 131 attempts, through its line merge function 134. to 
synthesize Infocmatioa about paragraph styles that may 
encompass successive lines. So, fear instance, if in a Une- 



TABLE 3 



New pagD— ftigoab the beginmng of a new p9igt aa Ifae rtonmifgrt 

Formal ov/o8 — toggles a won) caxpbasit style; in the pi e feK e d embocfimexU, recognized 

wotd empfaftsis styles aie uodetiiDe, boU&ce; and italici. 
Note-~«igDals the be^immg or ending of a fooCDOce or endoote. 
lUb— incficatefl that tbe next paregiapb begins at a O0w faorizoncal positkn on tbe p^ge. 
bdeot-Hndicales dut tbe next pam gt ap h is indented (or outdeiHed), dat is, it has a left 

and poosibty also right, maiigin diSereot than the pievious paragraph style. 
Retum — signals the end of a paragnph; inlxne^imilBd documents, return signals tbe 

ends of linea, which may (ben be meiged mto paragnq^ by a line merge 

function 134 (see below). 
Margin cfaaqge — indicates thai all new paragraphs will have left ancVcr right margins 

diflfereni than tbe previow paz^gn^ style. 
Leadiog cbange — tnriiratr^ that all new parB^vphs will have primary aivVor secondary 

ktadingw different than the previous paragraph style. 
JustiScation change — iTv<irTtt»4 tint all new paragraphs will have a justificatioQ (fifferent 

than tbe previous paragrqih style. 



For some of these codes, additional information is 
required besides the code itself. For this reason, it is desir- 
able that the scanner 110 contain an auxiliary 'Unfonnation*' 
field that holds information related to the last returned code. 
The information field for the code tab for exan^le, should ^ 
contain the tab position and the tab type (e.g., left, right, 
center); the information field for die format on/off code 
should contain the type of fonnat to be toggled (e.g., 
underline, italic, boldface). 

A. Format Analysis 33 

The fonnat analyzer 130 takes its input from the scanner 
110, scanning the entire document 10 to gather a list of 
**paragrq)h descriptor codes" that describe each paragrqp^ 
type in the document 10. Even befc»re this operation, 
however, the format analyzer 130 briefly scans the begin- 4^ 
ning of the document 10 in order to determine whether the 
document 10 is line- or paragraph-delimited. For this 
purpose, a simple function merely counts the number of 
lines read by the scanner, and the number of "long lines'* — 
those lines whose lengths exceed the expected boundaries of 
the document If greater than 2% of the lines read are long 
lines, then the document 10 is considered paragraph- 
delimited. 

FIG. 3 d^icts the q>eration of the format analyzer 130, 
once die delimiting typNc of the document has been decided. ^ 
As characters and codes are read in from the scanner 110, the 
format analyzer's accumulator engine 131 accumulates 
information about the document 10 in two forms: the 



delimited document a series of four lines are not paragnqihs 
in their own right, but are actually the lines of a single 
paragraph, the line merge function 134 will return the 
paragraph style that desoibes the paragraph comprised of 
those four lines. If instead the lines are each independent 
paragraphs (as, for example, in a heading block or a title), 
the line merge function 134 merely returns the paragrq)h 
styles that represent each line individually. In the preferred 
embodiment, the line me^e function 134 <m]y merges lines 
of paragr^hs that appear to be left justified. 

The line merge function 134 operates by testing pairs of 
lines to see whether they could be part of the same 
paragraph, based on a set of rules derived from the nunner 
in which lines of text are normally broken at the rig^t margin 
to form paragraphs. Where two lines ai^ear to be 
**mergable,'' that is, part of the same paragraph, die line 
merge function 134 determines the minimum and maximum 
right margins of a paragraph containing those lines. As more 
succeeding lines are "merged** to that paragraph, the mini- 
mum and maximum right margins will converge on the true 
right margin of a paragr^h comprised of those lines. When 
a succeeding line caimot belong to the paragraph, the line 
merge function 134 returns the margins, leading, and other 
infonnation regarding that paragraph, and begins searching 
for the next paragraph. The line merge function 134 is 
described in pseudocode in Table 4, below. 



TABLE 4 

Define the following variabtes and obyects: 
Paragraph = an otject ocrreGpooding to the curreotly analyzed paragraph of text, having 

Rij fctMhi and Ri^uMsot variables, which are die mmimiiTn and maxxnum right 

margnw possible for the paragraph, respectively 
First = an object """"p""'^*"g to a first line of text, having Lefl and Ri^it margins, a 

CmtKT point measured midway between Left and Ri^ a Secoodaiy lead^ 

(measwed w tbe vertical spacing between this line and the previous <joc\ a 

PomaU Sag (true if the text of tbe line is aH capitals), and a lustificatioo (left, 

right, or center). 

Second » on object like Rrat, oo t reapcoding to a seccod line of text which immediately 

follows the first line of text; 
bFoit = a Boolean mable, tme if First a the first line of text in Paragraph. (If; for 
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TABLE 4-oontinucd 



excmpk, Rnt is the tfaizd line of text in Psragnph, then ZoFiret = Fabe.) 
WsfdstnFint = the aumber of woids m Fnt, tieatiag woidc u acqucncea of lymbols 
fkiliiiiili^ by cpaoes. 

La5tRi^ = FtnUUght phis the Veagth of the first word in Seoood, treariog the word u 
the Icogest mrmrT of cbandBn which cazmot be bnAen acoovcfing Co Boghfib 
typeaettmg oonvemioafi (e^^ a sequence of chmcteis not iuctudiitg a space or 
hyphen). 

Iblerance = a variable <V>fini^ the petnuasible misaligmiieiit of the rij^ margin of 
Paragraph, to aoco^tnt for hunan or "T*^>wr^ error in breakiqg the lines of 
pangrapfa. In the prcfef red emfcodimeat Iblennce ia ^pfoximately 2 picas. 

MtnmunWnds = a variable A» fm"^ the fn;;nitnfn number of words in the first line of 
text in Paragr^sb, to piouipt '"^'glfyg In the preferred coabodinMirii, 
Minmum^fards is three. 

Hko, for die variables defined above. First and Second can meige Cie., they 
bekiQg to d» cunoK Paragraph) U (((liFint = Falae) and (FintXefk - SeoondLeft)) or 
((l^mt = IhK) and (FirstXeft <= Seooad.CcatBr) and (Secoodixft <= FintCeoter})) 
and ((IsFirst = False) or ((Swwt Secondary = FffstSecoodaiy) and (SeooolSeooodary 
>= 1))) attl ((SecoodRigfat - LastRight) <= Iblerance) md ((SecoodRight - RightMax) 
<^ Iblerance) and ((RigfalMin - LastRight) <= Tbkrance) and (FustJustificatioo = 
SeeoodJuBtification = Left) and (FinLPcrmat - SecoodPannat) and ((UFmt ^ False) 
or (WordshiFint MnmnnmWjids)). 

If First and Second can neige, dien Ptaagr^iih.R]ghtMni ~ 
mai(PafagTaph.mgfalMin, I^rstRight)* and Paragraph. R jghtMax s 
min(PBragraphJUgltMaXf LastRigliX aikl if (Paragraph RightMin > 
ParagrapfaJUgbtMax), then swep(P aj a gf ap h.R igjh TMin , PaiagrapbJUgfatMax). 



As each new paragraph style is returned in tttc paragr^ 
accumulator 153 (for paragraph-delimited documents) or the 
line merge functioo 134 (to line-ddimited documents), the 
style is entered in a xcmpomy paragraph style list 135. At 
the same time, the location of the start of the paragraph in 
the document that ootresponds to that paragr^ style, 
together with a cross-reference to the style list, is entered in 
a temporary paragraph style reference list 136. (Paragraph 
styles are added to the tenq>orary style list 135 uniquely — 
that is. duplicate styles are not added. Instead, if a duplicate 
style is detected, the cross-reference added to the paFagnq)b 
style reference list refers to the first occmrenoe of that 
paragraph style.) 

Entries in the temporary style list 135 arc termed para- 
graph descriptcv codes. Each paragraph descriptor code 
must contain sufficient infonnation to describe the layout of 
the cutrent paragraph. Usually diis information will include 
the left and right boundaries of the paragraph, and its 
primary aod secondary leading. In the preferred 
embodiment* slightly more infonnation ts ii>cluded about 
each paragrq)h: the paragr^ style may include a first line 
indent, a justification code (e.g.. the text of the paragr^h 
may be left justified, ri^t justified, or centered between its 
margins), and a fonnat flag that indicates whether the entire 
paragraph should be written in uppercase letters. This addi- 
tional information enhances the storage efficiency of the 
overall system, by eliminating many other otherwise neces- 
sary f<xmatting codes. 

Once the document 10 has been completely scanned, and 
the temporary style list 135 and paiagrai^ style reference 
list 136 have been built, the format analyzer 130 attempts to 
minimiTa the number of paragraph styles necessary to 
describe the document Line-delimited documents, in 
particular, will appear to have a large number of paragraph 
descriptor codes, because some lines will not have been 
merged by the line merge function 134, and because lines 
that are center or right justified will be recorded in the style 
list 135 as left justified paragraphs having odd margins. 
(Since most line-delimited documents have no format 
encoding other than spaces and horizontal tabs to indicate 
justification, the line accumulator 132 treats all lines as left 
justified, and records the actual boundaries of the text of the 
line as the margins of the paragraph.) 



To refine the temporary style list 135 of line-delimited 
documents, the fonnat analyzer 130 next attenq>ts, as shown 
in FIG. 3. to assign to each paragraph style in the style list 
135 its proper justification, through a paragr{q)h justification 
assignment function D7a. C^ie justification assignment 
function 137a is not called for paragraph-delimited 
docimients.) The justification assignment function D7a 
need only operate on those paragra(4> styles not merged by 
the line merge fuiK:tioD 134, because those paragraph styles 
are assumed to be left justified paragrqshs. The justification 
assignment is most easily and accurately aocomplisbed by 
constructing an alignment point list that contains all possible 
alignment points for each paragraph style in the style list 
135. (The aligrmient points of a paragraph style are its left 
boundary, its right boundary, aod its center.) The frequency 
of occurrence of each alignmeot point is tallied as part of the 
list-building pnxress. The alignment points of each para- 
graph style in the style list 135 are then matched against the 
alignment point list and a '^preferred"* aligrmient point list is 
built by entering into this list the matching alignment point 
from the aligrmient point list having the greatest frequency. 
Ties are generally resolved in favor of left justification, 
although where alignment point appears to be centered 
between the document's margins, tics are resolved in favor 
of centering; and where the alignment point is right justified 
at the document's right margin, ties are reserved in favor of 
right justification. As before, the frequency of occurrence of 
each preferred alignment point is tallied as pan of the 
list-bunding process. Hnally, the aligrmient points of the 
paragFG^ styles in the style list 135 are again matched, &is 
time against the prefexred alignment point list Hie matching 
alignment point having the highest/frequency indicates the 
preferred alignnient of the paragrz^ style. 

Next, for line-delimited documents only the tenqxrary 
style list 135 is again scanned by a final line merge function 
137^ to determine if any successive left justified paragraph 
styles can be **merged,** as before, to create a single para- 
gr^h style. The final line mcxge function 137fr will merge 
any juxtaposed left justified paragr^hs (lines) in the style 
list 135 that meet all of the criteria of T&ble 4, and for which 
the paragraph style that encompasses those lines (le.. that 
particular combination of margiDS. leadings, and format) has 
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already been encountered at some earlier point by tiie line 
merge function 134. 
If desired, further refinement of the tempoTaiy style list 

135 may be accon^>lished by a boundary realignment func- 
tion 13?c and a leading conforming ftinction 157^. The ^ 
boundary realignment function 137c attempts to realign all 
paragraph style boundaries so that they fall at regularly 
distributed points across the page. If the boundaries of a 
paragraph are at 11 picas (left) and 73 picas (right), for 
cxai^e, the bound^ realignn^nt function 137c might 
alter that paragraph style's boundaries to 10 picas and 75 to 
picas, respectively. The efifect of this function is to reduce 
the number of different paragraph styles, inqvoving storage 
efficiency and. In many cases, in^voving the look of the 
reproduced document. 

The leading conforming function 137t/ performs a similar 
role. It scans the entire tencqx^ary style list 135, altering the 
primary and secondary leading of each paragraph style to 
prcf ened norms — in the case of paragraph styles that appear 
to be body text (typically those having margins at or near the 
page boundaries, and consisting of three or more lines), 
primary and secondary leading are set to 1 and 2 lines, 
respectively; all other paragraph styles are unchanged, 
unless their secondary leading is greater than 2 lines, in 
which case it is changed to 2 lines. The result of the leading 
conforming function tyjd is generally a reproduced docu- 
ment having more consistent vertical spacing, and better ^5 
storage efficiency, as fewer paragraph styles arc required to 
represent the document 10. 

Next, the tenq>orary style list 135 is scanned by the 
secondary leading adjustment function 137^ to detect and 
delete high secondary leading, low frequency paragraph 30 
styles. These paragra{^ styles are infrequently used, but are 
distinct from other paragr^jb styles because their secondary 
leading is very large. Coimnonly, these styles occur at the 
first instance of a new heading, or a title block. To improve 
storage efficiency, the secondary leading adjustment fiinc- 
tion 137e deletes these styles, rercferences the paragraph 
style reference list 136 to an analogous paragraph style 
having a lower secondary leading, and inserts one or more 
*Wke-up** paragraph style references in the reference list 

136 to con^>ensate for the change in secondary leading. (The 
*^make-up** paragraph style references have a secondary ^ 
leading of one or more, and thus serve as place-holders 
between the current paragr^ and the previous one.) 

Finally, the temporary style list 135 is again scani^ by 
a paragraph style matching function 137^, to eliminate any 
duplicate styles that may have resulted froin the operation cf 4S 
the jHwious five functions 137a-137f . The style matching 
function also iQKlates the paragraph style reference list 136 
to accommodate any changes made in the style list 135. Hie 
final style list 135 and reference list 136 comprise die 
metafile's paragraph descriptor table 23 and paragr^h so 
descripCCH^ reference list 24, req)ectively. 
B. Lexical Parsing 

Returning now to FIG. 2, the scanner 110 is reset to the 
beginning of the document It, so that the lexical parser 120 
may use it to scan the document 10. However, the lexical 
parser 120 takes paragrq^h formatting infcvmation from the 
format analyzer 130, instead d from the scanner 110: as will 
be described below, the original documoit's 10 formatting is 
ignored in favor of the conq>act format infoimadon gener- 
ated by the format analyzer 130. 

The lexical parser 120 pcrfonns the function of sq>arating ^ 
the text of the document 10 into search terms (the words of 
the document that are not stopwords), metacodes (the 
punctuation, formatting, stopwords. and morphological 
transformation flags between search terms), and residuals 
(everything else). 65 

Referring now to FIG. 4, the output from the scanner 110 
and the format analyzer 130 are passed through the lexical 



parser's state engine 121. The state engine 121 accumulates 
strings of characters or codes that represent either potential 
search terms <^ the punctuation/formatting information 
between two search terms (this infmnation wiU be called a 
^^tacode string**). Preferably, the state engine 121 accu- 
mulates search terms and metacode strings in a term buffer 
and a code buffer. re^)ectiveiy (not shown in FIG. 4 for 
clarity). Also preferably the code buffer represents the 
punctuation/formatting preceding the term contained in the 
term buffer. The state engine 121 may accumulate more than 
one search term in a row — that is, it may accumulate a 
search tenn, process it (as described below), and then 
accumulate another search term, and so on — but it may not 
accunwlate and then process more than one metacode string 
in a row. (Since metacode strings represent all of the 
punctuation and fcrmatting infonnatioo between two search 
terms, by definition, no more than one metacode string exists 
between two search terms.) Together with the term and code 
buffers, the state engine 121 also maintains a position 
counter (not shown) ^nbkh is incremented as each new term 
or metacode string is read 

The state engine 121 is preferably constructed as a 
deterministic finite automation (DFA), a technique well 
known in tbc prior art, which transitions between discrete 
states when certain characters or codes are received from the 
scanner 110 or format analyzer 130. For example, as shown 
in FIG. 4, die state engine 121 is accumulating punctuation 
or fcnnatting information when it is in "state 0.** If it 
receives al^anumcric characters, it will send the already 
accumulated infcnnation on to be processed (as a metacode 
string), and then transition to "state 1,** where it anticipates 
reading a potential search teniL (The transition arcs joining 
the states shown in FIG. 4 are not shown, for clarity.) 

In the preferred embodiment, states 2 and 3 are special- 
ized fonns of state 1, which are designed to handle potential 
search terms having embedded or trailing periods (e.g.. 
-U.S.") or embedded q>ostrophes (e.g., **they*rc**), which 
would otherwise shift the state engine 121 back into state 0. 
It is {deferable that a search term include any embedded 
periods — periods that are surrounded on both sides by 
alphanumeric characters — and that if, and only If, an embed- 
ded period is present in a term, the term also include its 
trailing period This system most appropriately accommo- 
dates typical abbreviations, and enhances storage efficiency. 
It is likewise preferable that search terms include embedded 
apostrophes, to accommodate contractions. 

State 4 deserves special mention. As noted above, the 
lexical parser 120 ignores the paragraph formatting of the 
original document (its carriage returns, tabs, indents, margin 
changes, etc, and often spaces, as well) in favor of the 
compact, and possibly altered, paragraph fcnnatting gener- 
ated by the format analyzer 130. For this purpose, the sUte 
engine 121 checks the format analyzer's paragri^h style 
reference list 136 to determine when new paragraphs be^n. 
When the reference list 136 indicates that the beginning of 
a new paragraph has been reached, the state engine 121 
shifts into state 4. and ignores any spaces or paragraph 
formatting codes sent by die scanner 110 until fresh aiptia- 
numeric or punctuation characters are received, shifting it 
into states 0 or 1. However, state 4 presents an interesting 
problem: in line-delimited documents, the end-of-line 
delimiter and surrounding spaces will be ignmd, resulting 
in the possible concatenation of the last word from the 
ivevious line with die first word of the next line. For this 
reason, the state engine 121 inserts a special interim meta- 
code into the accumulating string of characters when the 
engine 121 enters state 4. The lexical parser* s interim filter 
(described below) will later remove the interim code, and 
replace it with zero, one, or two spaces, as necessary to 
represent apprc^viately the space between words within a 
sentence or across the end/beginning of a sentence. 



01/29/2004, EAST Version: 1.4.1 



5,704,060 



15 

Infomuition that is diaracterized by the state engine 121 
as a potential search tcnn is transfcmd to a set of tenn 
transfonnatioo functions 122, as shown in FIG. 4. The 
transfoamatioD functions 122 operate in the following 
sequence. First a decapitaiization function detennines 
whether the term in bufifer contains cqntal letters. If it does, 
the dec^italization function deopitalizes the term (Lc., 
convcfts it to all lowercase letters), and appends a recapi- 
talization metacode to the code buffer. In the prefeired 
embodiment, dierc are three recapitalization metacodes (see 
Table 2, above): caps, which causes only the terrn^s first 
letter to be c^italized (except that, for w<xds witti embed- 
ded periods, c^s also causes each first letter following a 
period to be a^talized); allcaps, which causes all of the 
tenn's letters to be c^italized; and spedalc^, which 
causes any of the first sixteen letters of the term to be 
capitalized. (When the deaq;>italization function adds a 
specialcaps metacode to the code buffer, it also adds a two 
byte flag to the residuals list 25. Each bit in the flag indicates 
whether the ccnesponding letter of the term is oqntalized cr 
not) 

Next, a depossessive function determines whether the 
term in the texm buffer has an ^^'s** suffix. If it does, the 
depossessive function removes the suffix from the term, and 
appends a possessive metacode to the code buffer. 

Next, a st(^ord removal function determines whether 
the term in term buffer is a stopword. If it is, the 
stopword removal function clears the torn buffer, and 
^spends a metacode representing the stopword to the code 
buffer. 

Next, a singularization function determines whether the 
term in the term buffer is a plural inflection. If it is, the 
singularization function converts the term to its singular 
lemma, and ^>pends a plural metacode to the code buffer. 
(Of course, ids function also accounts for the regular 
present tense — third person singular inflection ci verbs.) 
The singularization function of the prefened embodiment is 
designed for reversibility, not necessarily accuracy. Rather 
than treat it here, the singularization function is described in 
more detail in Section VH, below. 

Next, an -ed stemming function determines if the term in 
the term buffer is a regular past tense or past participle 
inflectional fonn (ie., it has an '^-ed** suffix). If it is, die -ed 
stemming function converts the term to its lemma, and 
appends a plusEd metacode to the code buffer. Like the 
singularization function, the -ed stemming function is 
designed pdmarily for reversibility, and not necessarily 
accuracy. It is treated in more detail In Section vm, below. 

Finally, and only if the -ed stenmung function did not alter 
the term, an -ing stenmung function determines if the term 
in the term buffer is a regular present participle inflectional 
form (Le., it has an "-ing** suffix). If it is, the -ing stemming 
function converts the tenn to its lemma, and appends a 
pausing metacode to the code buffer. Like the -ed stemming 
function, the -ing stemming function is designed primarily 
for reversibility, and not necessarily accuracy. It is treated in 
mere detail in Section VIDL below. 

The fully transfcvmed term (assuming it was not a 
stopword, and therefore has survived the stopword removal 
function) is then passed on to a term block oonQxesscr 124 
and block compressed. The details of the block compression 
process arc described more fiiUy in Section VI, below; for 
now, it is only important to understand that the term block 
cowpccssot 124 creates a compressed representation of the 
search term in the term buffer, and that only the first few 
significant bytes of this comfxessed representation will be 
stored in the metafile's w<Hd vector table 21. These first 
significant bytes are termed a dword (firom "dictionary 
word**). In die preferred embodiment, the lexical parser 120 
stores up to six bytes of the compressed search term in the 
metafile* s word vector table 21. If the con:4}resscd search 
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term contains more than six bytes, the lexical parser 120 
stores the extra bytes (a word completion code) in the 
metafile's residuals list 25, and then appends a resid meta- 
code to the code buffer. Of course, as mentioned above, the 
compressed terms art only stored uniquely — that is, if the 
term is already found in the word vectcv table 21, it is not 
added again. In any case, however, the state engine 121 adds 
the term's current fuzzy position (as measured by the 
position counter) to the term's position list in the word 
vector table 21, and then increments the position counter. 

0 Note diat the position list for each seardi term in the word 
vector table 21 is maintained in an ascending order. 

Information that is instead diaracterized by the state 
engine 121 as punctuation/fmnatting information is passed 
to a set of code transfonnation functi<xis 123, as shown in 
FIG. 4. The transformation functions 123 perform general 
clean-up operations on the metacode string in the code 
buffer. The first function to operate is an interim metacode 
filter, which scans the metacode string in the code buffer for 
interim metacodes added by the state engine 121 while in 
^state 4," as described above. The interim metacode indi- 

0 catcs that the state engine 121 ignored the document's 
original formatting codes, such as carriage returns, tabs, and 
spaces, while it accumulated the current metacode string, an 
event that typically occurs when successive lines of a 
line-delimited docmnent 10 have been ^'merged** by the 
format analyzer's 130 line merge function 134 into a single 

^ paragraph. The interim filter checks the metacodes sur- 
rounding die interim metacode and replaces the interim 
mmcode with (1) nothing, if a hyphen metacode precedes 
it; c^erwise (2) two space metacodes, if end-of-sentence 
punctuation metacodes (e.g., a period, exclamation point 

0 question mark, or colon, optionally followed by closing 
quotations, parentheses, or brackets) precede it, and the first 
character of the next word (the term currentiy in the term 
buffer) is capitalized; and otherwise (3) one space metacode. 
In this manner, the interim filter axtcmpis to place the correct 

3 number of spaces between the last woid of the p^vious line 
and the first word of the following line. 

Next, a space filter replaces any string of n or fewer 
contiguous ^oe metacodes with a single space metacode. 
The purpose of the space filter is to convert documents 
having irregular horizontal spacing-the most conunon 

° examples of these are line-delimited documents that have 
been left-and-right justified by inserting extra spaces into 
each line so that the right nurgins of each line are flush — 
into text having a more regular spacing. The result is a 
rqroduced doounent having more natural looking text, and 

s increased storage efficiency. In normal operation, it is pre- 
fened that n, the space filter threshold, be five spaces. Larger 
sequences of spaces probably represent an intentional effort 
by tbc author of the document 10 to place text at a certain 
position, as in a table. 

0 Next, a special punctuation filter translates some punc- 
tuation sequences in the metacode string in the code buffer 
into other, preferred punctuation metacodes. In the preferred 
embodiment, die special punctuation filter (1) translates 
sequences of two to four hyphen metacodes into a single 

J "emdash" ( — ) metacode; (2) translates combinations of 
three alternating spaoes and periods into a single **three 
period ellipsis" (...) metacode; and (3) translates combi- 
nations of four alternating spaces and periods into a single 

*tour period ellipsis" ( ) metacode. 

Finally, a repeat filter scans the metacode string in the 

° code buffer for rq>eating series of n or more metacodes. (In 
the preferred embodiment n is five.) If found, the repeat 
filter replaces die scries widi a repeat metacode followed by 
the metacode to be repeated. The repeat filter also stms in 
the metafile's residuals list 2S a single byte indicating the 

$ number of times the HD^acode is to be rqpeated. 

The fully filtered metacode string is then block oom- 
jH^essed by a metacode block compressor 125 (die details of 
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the block compressioo process are described in Section VL 
below). Next the lexical parser 120 increments the position 
counter by the number of byte in the block compressed 
string, to account for the *'positioas** occupied by the code. 
Finally, the block ccMxqiressed metacode string is passed to 
a Huffman conqircssar 126. which con^sses the metacode 
string and then appends the conqsessed string to the meta- 
file's code list 22. (The Huffman compressor 126 is also 
described in more detail in Section VI. below.) 

Infonnation from the document 10 that is not character- 
ized by the state engine 121 as a search term cr punctuation/ 
formatting infcHination Is sent direcdy to the metafile's 
residuals list 25, as shown in FIG. 4. In the preferred 
embodiment, the only information sent direcdy &om the 
state engine 121 to the residuals list 25 is the t>pe of word 
emphasis that accompanies a format on/off metacode. The 
state engine 121 sends a single byte code to the residuals list 
25 that represents, in its lower three Uts, which types of 
word emi^asis (underline, boldface, and/or italics) are to be 
toggled. In other embodiments, of course, it is possible that 
the state engine 121 could pass on to the residuals list 25 
information such as embedded graphics, hypertext links, and 
more advanced formatting hifcrmation. 

For illustration purposes, TaUe S below shows the output 
of the lexical parser 120 f<^ a shcHt sample text The 
paragraph descriptor codes are listed following the 'TDC* 
tag; search scans appear as normal text, and mctacodes are 
bracketed. The underscore sq)arating search terms indicates 
that there is a space between the words, but simple spaces 
between words are never output by the lexical parser 120, 
nor stored in the metafile 20. Note also that the spaces 
between stopwords and search terms, or other stopwords, are 
also not stored. These spaces are instead regenerated during 
the expansion process (described below), whenever two 
words would otherwise be adjacent. 
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table 21 having a unique list of up to the first six significant 
bytes of each block compressed search term in the document 
10, together with the fiizzy positions in the document 10 that 
those terms occupy; (2) a code list 22 that contains Huffman 
encoded, block con^ressed metacodes. which describe, in 
order of occurrence, all of ttie punctuation, forroatting, 
stopwords, and moxph(rfogical transfcxmations that occur 
between every search term; (3) a paragraph descr^tor table 
23 and paragraph descriptor reference list 24 that togetho- 
de scribe the paragraph format (margins, leading. 
justificati<Hi, first line indent, and capitalization) of every 
paragra;^ in the document 10; and (4) a residuals list 25 that 
contains, in order of occurrence, miscellaneous information 
about the document 10 such as word entasis types. icpctX 
counts, and word completiOD codes. 



IV. The Archive Structure 
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The ardiive structure 30 created by tfiis invention is 
intended to reside in fixed storage, such as on a hard disk 
drive or CD-ROM. It comprises eight separate substructures, 
stored as separate files. Some of these substructures may be 
merged together in a single file located on the storage device. 
For purposes of this discussion, howeva, it is easier to 
consider eadi substructure as a sq>arate file. FIG. 5 shows 
the substructures, or files, of the archive 30. Table 6, below, 
describes the contents of each file. 



TABLES 



Ad Unerpected Party 

la a bob in the ground &eze lived a bobbit Not a nasty, dirty, wet hok, fiUod with tfae 
ends of wonns and an oozy oneU, nor yet a dry, bare, saody bole with ootfaiag in it to 
sit down on or to eat; it was a hobbit-bole, and that mean oomfibit. 
[FDC:1 L.'020 R:1S0 F:000 FL:1 SLK) FOK) Center) 
[cap] ian] (cap] [i^u^] unexpect [cap] party 
(PDC:2 L.O20 R:130 F:000 VL:l SL:2 FOK) Uft) 

{cap] (in] (a) hole [in] [dte] ground [tfam] [plv^] Uve [a] bobbit [.) [cap] [not] [a] 
nasty [.] dirty [.] wet^Ie [.) [phiaEd] fill [with! [tt»l [phnt] end [of] [phnJ] 
wonn [and] [an] oozy_sniefl [,] [nor] yet [a] dry [,) bare [,] saody..J»k [wttfa] 
[phising] ooch [in] [it] [to] sit_dowD (co) [or] [to] eat [;] [it| [was] [a] bobbit [•] bole 
[,] [and] [tfaal] [phsal] xnean^oomfbrt [.] 



At the end of the lexical parsing and fc»mat analysis 
processes, then, the metafile 20 contains (1) a wcHd vector 

TABLE 6 



1. Archive inftinnatioQ file 31— containfi in archive infbnnatioa block 3 la which 
ncxxda geneial infonnatiaD aboiM the archive 30, luch as tfae number of 
finnnnniCB fiKmd in tfae archive and compiescion ratios; and a list of **free space 
aurfcen** 31b whkb identify free spaces in index file 34. 

3. RegioD file 32~ocaiains a list of pointers vfaicb identify the b«ginning of each 
(VxntTmit in die dmimmT file 33. 

3. Docunvatf file 33-^cootains the donwimfs of tfae aicfarve, stoied as oompresaed 

4. Index tile 34 — ocntains dncumwt reference vectors for eveiy search tarn m the 
archive dictioniry 36, and dioae search terms in tfae standard dactaooary 33 d&t 
are fioond in die archive 30. 

5. Standud dktkxury file 35-^ooiitains a list of unique search terxoa» stored is 
dwonb, comnsoly found in documents 10. In the preferred ^iiibwlimenl, the 
stBiulsrd dictionary 35 ccntaina apt s ujiim atety 3000 commoo business, legal, 
and tediucal tenna, such as "inierestr "msn," "new," •leaaon," "said,** 
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TABLE 6-contuiued 



"itatrnieot," sod ""within." 

6. Archive dictnoary file 36— domains a list of all unique search terms (other than 
those already id the staiMki J dictiooary 35), ftom) as dwords, found ia die 
documents ori*tiifwi iq the axduve 30. 

7. SlBulard dkrtknaiy Imk fik 37--cocitaiM a points 
search term in the standard dicttooary file 33. 

8. Archive dictionary link &le 38 — oootains a pointer to the index file 34 for each 
search term in the archive dictioiuDry file 36. 



V. Adding a Metafile to the Archive 

RG. 6 ilhtstrates the procedure for adding a iDCtafile 20 
to the archive 30. Briefly described, the metafile's word 
vector table 21 is matched against the archive's two dictio- 
Darics 35, 36, aod then compressed and appended to the 
document file 33. The rest of the metafile 20 is thea 
compressed and q)peaded to the document file 33, as well. 
Next« the document's serial number is added to the archive's 
index file 34 for each search term in the word vector table 
21. Finally, the ardiive information file 31 is iqpdated to 
reflect the addition of the new document 10. 
A. Storing the Metafile in the Document File 

Before the metafile 20 representation of a document 10 is 
added to the archive, a short, blank document header 33a is 
q>pended to the document file 33. The header 33a will later 
be filled with information about the document 10, but 
because this information is not yet available, the header 33a 
is written sin^^y as a placeholder. 

Next, a dictionary matching function 141 compares the 
dwords stored in Ae metafile's word vector table 21 to the 
list of dw<xds stored in the standard and archive dictionary 
files 35, 36. If a match is found, the dictionary matching 



the one before it Because the dictionary references were 
maintained in an ascending order, the average magnitude of 
the new, differenced references will be greatly smaller than 
that of the original, absolute references. During this replace- 
ment process, therefore, the differencing comjH'essor 142 
determiDes the n^f"*"^"™ number of bits required to encode 
the largest differenced reference (the "maxbits" of die list <^ 

20 references). 

As shown in FIG. 6, the differencing compressor 142 also 
operates on die fuzzy position list stored with each search 
term in the word vector table 21. perfonning die same 
function of r^ladng each fuzzy position in the list with the 

^ difference between it and the one before it, and determining 
the minimum number of bits required to encode the largest 
differenced position in Ihe list (the **maxbits** of the list). 
Because the number of positions in each list (the **li5t 

^ county must also be stored along with the list, however, the 
differencing compressor 142 uses an opdmized method to 
encode the count and the positions togetfier. Table 7, bdow, 
describes the system foUowed in encoding the fuzzy position 
lists. 



TABLE? 



1. If the list oontaina only coe reference, magnimrin leas than 32768. encode as a 
T bit, foUowed by the poaito stored in fifieen bits. 

2. If die list count is three or to, encode as a **0r bit, foUowed by the count stored 
in two bits, then the maxbits stored in faw bits, then the list of differenced 
references stored in luubtts bits. 

3. If the list count is between four and ten, encode as duee **(r bits, followed by the 
count minus three stored in throe bits, then the maibits stored in four bits, then 
the list of differenced lefereoces stored in maxbits bits. 

4. If the list count is between eleven and 1033, exKode as six *Y) " bits, foUowed by 
the count minus ten stoied in ten bits, then the ntaxbita stored in four bits, then 
the list of differeoDed leferenoes stored in mazfoils bits. 

5. In all other cases, encode as the coual minus 1033 stored in thirty-two bits, dien 
the loazbitB ctoied in four bits, then die list of difEerenoed references stored in 
maxbits bits. 



function 141 r^laoes the dword in the word vector table 21 
with a reference to the dictionaiy entry containing the 
dword. If a match is not found, the matching fimction 141 
adds the dword to the archive dictionary 36, and then 
replaces the dword In the word vector table 21 with a 
reference to the new dword in the dictionary 36. During the 
dictionary matrhmg process, the word vector table 21 is 
maintained in a sorted, ascending order, according to the 
magnitude of the dictionary references. Furthermore, dic- 
tionary references are preferably four bytes long, giving the 
dictionaries a combined potential range of over four billion 
words. 

The partially transformed word vector table 21 (now 
containing references to the dictionaries 35, 36 instead of 
dwords) is then passed tfaroug|i a sequential differencing 
compressor 142. The differencing oons^Htssor 142 replaces 
each dictionary reference with the difference between it and 



so 

The differencing compressor 142 then creates the 
archive's dictionary reference/position vector table 33b by 
writing in the document file 33 each differenced dictionary 
reference (stored in maxbits) and its position list (stored as 

ss described above in Table 7). 

To increase search efficiency, however, it is preferable that 
the dictionary reference/position vector table 336 be written 
in two partitions: the list of dictionary references and 
positions for terms found in the standard dictionary 35, 

^ followed by the list of dictionary references and positions 
for terms found in the archive dictionary 36. In the preferred 
embodiment, then, the se<^uential diffcrendnc compressor 
142 creates two separate dictionary reference/position vec- 
tor tables 336, one for each dictionaiy 35. 36. and calculates 
the maxbits for each set of differenced dictionary references 

65 separately. 

Next, the code list 22 and residuals list 25 are appended 
to the document file 33. as shown in FIG. 6. Because the 
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code list 22 is already fully coiiQ>ressed, It needs no further 
processing before final storage in die archive 30. Likewise, 
because the residuals list 25 is likely to contain highly 
random, and therefore poorly comprcssihle, infOTination» 
and is in any case a relatively small structure, it need not be 
coroprcssed. Of course* in an implementation of this inven- 
tion in which nuich information is stcHcd in the residuals list 
25 (such as hypertext linking or embedded gr^hics), it may 
be desirable to process the residuals list 25 with a compres- 
sion algorithm before storing it in the arduve 30. 

Next, the metafile's paragraph descriptor reference list 24 
is passed through an ad^^ve variable-rate encoding com- 
pressor 143 and appended to the document file 33. The 
ad^)tive variants of variable-rate compressors, such as the 
Len^-Ztv algorithms, arc well known in the art; in^lc- 
mentations erf tfiese algorithms are publicly available on the 
Internet The pr^cnred embodiment uses an adiqptive Huff- 
man variant of the Lempel-Ziv aigorithnt based on the well 
known public domain in^lcmentation coded by Haruhiko 
Okumura and Haruyasu YoshizakL 

Next, the metafile's paragraph desGr4)tor table (POT) 23 
is passed through a bitwise con^ressor 144 and <q>pendcd to 
the document file 33 following the paragraph descriptor 
ftfcrencc list 24. The bitwise conq>ressor 144 simply stores 
each paragraph descriptor code contained in the POT 23 as 
the oon^ct stnicturc described below in Table 8. 



10 
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for each new document To facilitate inunediate access to a 
given document by number, the archive's region file 32 
stores a sequential list of pointers to the beginning of each 
document's header 33a in the archive's document file 33. 
Thus, once a docimicnt's archive representation has been 
coziq>letely written into the document file 33, the location of 
its header is ^^pendod to the region file 32. 
B. Indexing the Metafile 

To enhance keyword searches of die archive 30, an 
inverted index file 34 is kept which, as diagrammed in FIG. 
5 and noted above in Table 6, contains document reference 
vectors for those search terms in the standard dictionary 35 
that are found in the documents in the archive 30. and for ail 
of the search terms in die archive dictionary 36 (which, by 
definitioD, are found in documents in the archive 30). These 
reference vectors are sorted lists of serial numbers indicating 
which documcnt(s) contain a particular term (together with 
a count rccOTd diat indicates how many documents contain 
that tarn). To speed access to die index file 34, the standard 
dictionary link file 37 and die archive dictionary link file 38 
contain pointers to tije index file 34 which parallel the dword 
entries in their respective dictionaries 35, 36. Thus, as shown 
schematically in FIO. 6, each link pointer in the link tiles 37, 
38 indicates the position of the document reference vector 
for diat term. 

Because documents 10 will be periodically added to the 
archive 30, die document reference vectors in die index 34 



TABLES 



1. Left umpa—out byte, i t yicMJiti i^ the dirtange torn fee bft edge of ttie page 

measured in 20tfas of an inch. 
3. Pij!* marggy— byte- inin a mti ng the digtaiice from right edge of tt» P»ge 

measured in 20th8 of an inch. 

3. Pint line iDckot--oiK byte, tepnaet«ii^ the distanoe fitm^ 
p^ge measured in TOtia of an inch- 

4. Primary badii«— four bits repreaeatixu the vextical space he«weeD lines in a 
pantgraph, nwasuied in eighths of a line, based al one line. For eiample, a 
primary leading of zero iuikatea single spacing; & primary leading of four 
iocficaies oi»-flzad-a-batf line spacing; a primary kading of eight inrtiratffs double 
spacii^g. 

5. Wfwto y l«^g — four bite fcpreaentiny the number of blank lines between 
paragrspto. A secoodsry kading of zero indicaks that the paragraph be^ on 
tbe sanv liw as die preceding pai agraph began (as in cohams and tables). All 
atixx values are Dvasured between the last line of tbe preceding paragraph and 
tbe 

first lioB of the cuncat one. 

6. Jiu li fi LMtio o t vn bits iTv<i«»rmj whether die paragraph is kfi, right, or center 

7. RMmat— one bit '"^Mt^^ whedier the paragraph is displayed in all capital 
ktters. 

8. Reserved— five bits. 



Finally, die document header 33fl is rewritten, diis time 30 
containing (1) the fuzzy position of the last search tcnn in 
the document 10; (2) die maxtnts size for the differenced 
dictionary references in the two tables 336; and (3) die 
distance, in bytes, from the start of the first dictionary 
reference/position vector table 336 (for die standard dicdo- 55 
nary 35 ) to the start of die second dictionary reference/ 
position vector table 336 (for the arcbive dictionary 36). In 
addition, because it is often desirable quickly to provide die 
user widi a list of citations to documents in the archive 30, 
the document header 33a contains a short citation, stc»ed as 60 
plain text, which identifies the document 10 (usually by title 
and/or author). 

At this point, the document 10 is assigned a serial number, 
so diat it may be rapidly retrieved, and also easily referenced 
by die archive's index file 34, which wiU be discussed 65 
below. The serial numbers of documents in the preferred 
embodiment arc numbered from zero, and increment by one 



will grow randomly. In addition, new document reference 
vectcrs will be added to die index 34 as new terms arc 
encountered in documents 10. It is therefore preferable dial 
the index 34 be structured as a free hei^, a structure well 
known the prior art A list of free space markers 316. which 
indicate &ee regions within tbe index heap 34, is stored at 
the end ardiive*s information file 31, as shown in FIG. 5. As 
new reference vectcrs must be added to die index 34, they 
are allocated space marked by the fircc space marker list 316 
first; but if no space is available within the he&p 34. then the 
reference vectxx is appended to the end of the index file 34. 
Similarly, when an existing reference vector must grow 
beyond its allocated space (because a new document refer- 
ence must be added to it), it is moved to the first accom- 
modating available space within the heap, or to the end if no 
such space is available, and its original location marked as 
free in die free space marker list 316. Of course, whenever 
a reference vector is added or moved, die pointer to its 
location must be updated in die relevant link file 37. 38. 
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As shown in FIG. 6, therefore, a linldng function 145 associated with each substring. For speed, it is desirable that 

scans the dictionaiy references from the archivc*s dictionary the input string be scanned from left to right for the longest 

referenoc^sition vectcH* table 336. The Unking function matching substring in the block list 300; when a match is 

145 adds new pointers to the link files 37, 38 for new terms found, the code associated with that substring is appended to 

in the archive 30 {Lc, those terms whidi were not already 5 output string, and scanning begins again at the input 

listed in the dictionaries 35, 36), and updates the document character following the matched substring. Fcx* example, 

reference vectors for each term, both old and new, by adding shows the compression and deconqression process 

to each reference vector the serial numbo- assigned to the ^ "*P"^ **thcnnocouplc." The longest substring in 

current document 10. hUxk list 300 matching the b^inning of "thermocouple" 

In the preferred embodiment, the document reference lo substring ''the"; its code (154) is therefore appended 

vectors in the index file 34 are passed through the sequential ^® ^ scanning begins again at the letter 

differencing con^ssw 142, using the same encoding in the input string. The process is then repeated until the 

scheme described for encoding position vectors (see Table 7, "*P"' has been scanned. Decompression is 

above), before storage. The differencing conqircsscr 142 not pcrfonned by sinqdy replacing each code with the substring 

only iniproves the storage efficiency of the index 34, but also 13 il rqwesents. 

makes it possible, in many cases, to add a document refer- hVxk list 300 is generated by scanning a very large 

ence to an existing reference vector without increasing the ^ conunon input strings to determine which substrings 

size of the vector beyond its allotted space. oonmK)nly repeated in the list The most corrmion 

C. Miscellaneous substrings are then ranked by value, where a substring's 

Once the metafile 20 has been added to the archive 30 and 20 value is e<iual to its frequency of occurrence multiplied by 
all its terms indexed and linked, the archive's information length in bytes. The highest valued substrings are then 
file 31 is updated to reflect the number of documents stored placed in the block list 300. In addition, to ensure that every 
in die archive. Information about die archive 30, such as the possible input string can be encoded by the block list 300, 
total size of documents stcred in it, cr the total number of hlock. list 300 must also include every individual char- 
words processed, may also be stored, and updated, in the 23 ^ ^ alphabet of possible characters in the input 
archive information block 31ii of die information file 31. strings. 

As an alternative to adding a metafile 20 to die archive 30, Abetter selection d substrings for die block list 300 can 
it is also possible for die metafile 20 (or several metafiles) to ^ niade by taking into account die complementary pbe- 
be compressed as described above, but not added to die nomena diat (1) when a long string is selected iot member- 
archive 30. In tills alternative embodiment of the invention, 30 ship in die block list 300, shorter strings already in the list 
die m^afile's word vector table 21 is not matched against which are substrings of the longer string, will have a 
die archive dictionary 36. Ratho^. die search terms in die lower value because die longer string will be now be used 
table 21 are matched against only the standard dictionary 35, rqplaoement instead of die shorter strings ; and (2) when 
and those terms found in the standard dictionary 35 are ^ short string is selected for die block list 300, longer strings 
replaced by references to the standard dictionary 35. In diis 35 already in die list 300, of which the sbocter string is a 
alternative embodiment, then, the coitqiressed metafile substring, will have a lower value because die longer strings 
becomes a '^transportable'* version of the document 10, r^laoe an input string which can already be more 
which can be transferred and used by any system or user efiSdendy encoded, in part, by die shorter string. Because of 
which has access to the standard dictionary 35. these phenomena, it is likely that for any reasonably sized 
VI m^u ^ '^^^ optimal selection of substrings for die 
VI. Block compression block list 300 is impossible to calculate in reasonable time. 

In parsing a document 10 to create a metafile 20, die However, a near optimal sohition can be found by iteratively 

lexical parser 120 en^iloys block conqressors 124, 125 to rescanning die list of ii^t strings, using die block list 300 

reduce die size of words stored in the archive dictionaries developed in Uie previous iteration. 

35, 36, and to reduce die size of mctacode strings stored in 43 ft is also desirable to use a second-order block list (a 

die mrtafiie's code list 22. Because die block oonqiression **biock tablc^, having a separate Ust of substrings for each 

technique is not known in die prior art, diis section wlU "position" widiin die iiqjut string. That is, during con^rcs- 

discuss die technique in detail sion die input string is matched against a first block Ust; after 

Block compression is a statistically modeled replacement its first substring is removed (and its code appended to die 

encoding technique diat is similar in many respects to so ou^ut string), die remainder of the input string is then 

known replacement encoding techniques such as Shannon- matched against a second block list (not shown in FIG. 7), 

Fano and Huffman encoding. (See Huffman, David A., "A and so on. The process is repeated using different block lists 

Mediod for die Construction of Minimum-Redundancy until die entire input string is encoded, or until the block 

Codes." ftocccdings of die LR.E., September 1952.) The com|Hessor mns out of lists. At dial point die last list can be 

Shannon-Fano and Huffimn schemes leplaoe die individual ss used again and again until die iiq>ut string is exhausted, 

bytes erf a string widi codes ^^ose lengdis arc proportional FIG. 8 describes d>c blocklist generation procedure of die 

to die frequency of occurrence die individual bytes. Block preferred embodiment, which uses secondnoider block lists 

oon^jression, by contrast, replaces commonly occurring generated by die iterative technique described above. Begin- 

substrings of a string widi fixed lengdi codes. Where die ning at steps 200-201, die first block Ust is initialized 

fixed lengdi code is byte- or word-sized, bodi con9>ression 60 (cn^) and a pass counter is set to 1. Also at stq> 201 die 

and decompression are extremely fast because partial byte values for aU substrings in die current block list m reset to 

operations are not required. zero. The Ust of input strings is tiicn initialized (reset to die 

The substrings to be replaced are stored in a block Ust 300, first string) at step 202. For die fir^i block Ust on die first 

shown schematicaUy in die middle of FIG. 7. Compression pass, stqw 203-206 are skipped; die initial block Ust is 

is acconq)Ushed by scanning die input string for substrings 65 generated by inserting into a temporary string Ust, at step 

found in die block Ust 300, and constructing an output 207, every left-handed substring <rf each input string. (The 

(conqyressed) string con^wscd of die fixed lengdi codes left-handcdsubstringsof die word "apple," for example, are 
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"a " "ap," "iq>p " **apph'* and "apple." Because it is unlikely described above) that would potentially be slOTcd in the 

that very long substrings will ultimately be selected for the archive's dictionaries 35, 36; and a list of several hundred 

block list, it is practical at this stage to omit to add substrings thousand non-unique metacode strings that would poten- 

beyond a threshold length. For exanq)le, when creating a tially be stored in the code lists 33c of documents in the 

block list to consprcss Hn^ii^A" words, very rarely will a 3 archive'i document file 30. These U^s were used to generate 

substring of eight or more letters occur with suflSdent the block tables of the term block compressor 124 and 

frequency to warrant mcmbcrshq> in the block list Longer metacode block comf^essor 125 of the preferred embodi- 

substrings can therefore be safely ignored.) Substrings arc meat 

entered into the string list, and their values (frequency of Conqxession efficiency for the dicti<Hiaries approaches, 

occurrence mult^>licd by length) tallied, during step 207. jq average, that of the Huflhnan tcdinique. However, while 

When all of the input strings have been processed, die block compression produces longer codes than the Huflfman 

substrings are sorted by value, and the highest valued technique for short input strings, it produces smaller codes 

substrings entered in the current block list, at step 212. For for long input strings^ for fix^-length 

the first pass, step 213 is ignored (because there is no storage, such as m the arduve s dictionaries 35, 36. 

previous pass to compare with), the pass counter is incre- In addition, block impression produces anoutpm string 

^nted at step 211, and the process repeats at step 202 by t^ving a smaUcr number of mteipal 'Mmts (the fixed length 

initializing^tting to the first stride lis7of input <^«*f *°P^ ^"^^ 

tT7~ ^ *^ twelve-letter input string such as **theniK)couplc" (see FIG. 

• . , . ^ . . ^ . 7) might transform to an output string having five codes. 

For the second and later passes, eadi input sfring is first q^j^ ^th the Huffman encoding technique, which 

tested, at step 204, agamst the current Wock hst (which was 20 ^oukl produce an output string measured in bits, having 

generated in the i^cvious pass). If the block list contains a pahaps thirty-five to forty-five bits, or at best in encoded 

left-handed substring of the input string (e.g., if the input letters, in which case the Hufl&nan code has twelve letters 

string is "apple,** and the current block list already Includes {(tie same as the input string). Block oompressiott is thus 

the string ^'app,** step 204 evaluates to yes), then the value ideal for stcning the metacode strings between search tcnns. 

d the longest matching substring in the block list is 23 because the number of **fuzzy positions** required by the 

incremented, at st^ 208. (In this way, the substrings of the compressed metacode string is minimized — improving the 

current block list are valued prefoentially over all other accuracy of position information stored in the metafile's 

substrings; this procedure also speeds the block list genera- word vector table 21. 

tion process.) If, on the other hand, die current block list did Finally, block conq>ression achieves higher compression 

not contain a left-handed substring of the input string, then 30 for strings fonned of long but common substrings — such as 

all of the input string's left-handed substrings are added to words fmacd of long but common English prefixes or 

the string list, as before, at st^ 207. (Note that the tenqxirary sufiSxes — at the expense of decreased conqnession for 

string list must be flushed at the beginning of each new pass, complex, but usually shorter, strings — such as nonsense 

at st^ 201.) At die end of this process, the values of all words, codes, and abbreviations. Thus, search terms such as 

substrings in the ten^rary string list, and from the current 33 anthropology and determination block compress more e£fec- 

block list, are ranked at step 212, and the highest valued tively than search tcnns such as int*! and xj4ti. This a^>cct 

substrings, from either source, are placed in the blodc list At of block compression is htg^y advantageous for keyword 

step 213, the block list is evaluated to determine if it has storage structures such as the dictionaries 35. 36 of the 

changed from the last pass (Le., whether any substrings were preferred embodiment, where die entire length of a common 

changed at step 212). In practice, it is not certain that this 40 keyword must be stored, to distinguish it from other words, 

iterative approach wUl converge on a single block list; for but where only the first few characters of a nonsense word 

this reason, it may be necessary at stq> 213 to determine need be stored to distinguish it from others. For exanq>le, the 

merely if the block list has substantially changed; but in block compressed strings for anthropology and anthr(q>o- 

generating blocks for both die dictionary block conq]ress<7 logical probably differ in the fifth ch- sixth bytes; similarly. 

124 and the metacode string block compressor 125 of the 45 the block compressed strings for xr4ti and xr4tzx also 

preferred embodiment, the block lists tend to converge on a probaUy differ in the fifth or sixth bytes. Thus, the same 

single solution within five to fifteen passes. number of bytes arc required to distinguish the two types of 

When the block list finally remains unchanged at step 213, words from other, similar words--a result that is also ideal 
the list is then altered to ensure that it contains at least every for fixed-length stCTagc structures, 
individual letter from the alphabet of all possible input so The number of substrings in each block list, and the depth 
strings, at step 214. (For example, in the block lists for words of the block table (Le., the numba of lists), is a matter of 
in the dictionary of the preferred embodiment, the block lists design choice which will depend on die type of input strings 
must have the single letters a-£, the digits 0-9, tfic period, to be encoded. The block table for the torn block compres- 
and the apostrophe.) The process is then repeated, as before, sor 124 has, in die ptc£are<\ embodiment, six block lists, and 
until at stq> 215 the last block list is reached. (When 55 each list has 256 substrings — which eliminates die need for 
generating the second or higher block list, however, it is partial byte <^>eration$, since each fixed length block code is 
necessary to remove from eadi input string the substrings of therefore exactly one byte. Hie block table for the metacode 
previous block lists. So. for example, whae the input string string block compressor 125 has, in the preferred 
is *'apple,** and the first block list has the substring *'appj* the embodiment, four block lists, and each list has 512 sub- 
input string becomes "le.**) 60 strings; the odd bit size (9 bits per code) has no effect on 

The list of input strings must be large, and suflBdently speed because, as noted above (and described below), the 

representative of all iiqHit strings to be encoded in practice, metacode string block compressor 125 is coupled to a 

to generate effective block Ubles. In the preferred Huffman con^iressor 126, which already performs partial 

embodiment, the lexical parso" 120 was used scan a 70 byte input and output functions for the metafile's code list 

megabyte corpus of pure text generating a list of ^Tproxi- 65 22. 

mately 100.000 unique words (after decapitaiization. stop- The Huffman compressor 126 is based on the well known 

word removal, and morphological transformation, as static Huffman compression technique. In static Huffinan 
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compression, each symbol in an input alphabet is replaced nouns, names, and other words which must be accurately 

with a unique code whose size in bits is inversely proper- respelled, a fully reversible singularization function is 

tional to the frequency of ocourence of the replaced symbol hig^y desirable. 

Thus, for example, in a Hu£Fman code for English words, the Clearly a dictionary-based function would suffice. Only 
very common l^tcr x might be rq>laoed by a very short s words in the dictionary of plural words would be singular- 
Huffman code, say 010; less common letters such as x might izcd\ to pluralize them again, the function need only locate 
be replaced by a long Huffman code such as 00110110001. the plural in the dictionary, given the singular fomL 
In Ihc preferred embodiment, the Huf&nan codes r^lace the However, dictionary-based methods are cumbersome, 
block codes generated by the block compressor 125. Thus, requiring at best a hashed search during the critical inner 
the Huffinan compressor 126 will rq>lace a very common lO \oop o[ the lexical parser's state engine 121, and likely a 
block code, say one representing the substring correspond- \argt dictionary as well. Moreover, a dictionary-based fiinc- 
ing to a period, two space m^aoodes, and a caps metacode— tion would be unatrfe to handle plural proper nouns and other 
which occurs at the end of OKKt sentences — ^with a short specialized woftls, such as jargon and abbreviations, corn- 
Huffman code. A less common block code, for exanqxle one monly found in business, technical, and legal documents, 
representing a right parenthesis and a space metacode, 15 -j^ prefored embodiment uses a partiaUy rule-based, 
would be replaced wife a longer Huffoian code. To furtha partially dictionary-based singularization function that is 
improve st<M^e ^^©^(7, each Wock list in the metacode reversible. The singularization function divides woids 
block compressor s 125 block table is assigned a separate ^ esndings into sevml categories of common 
Huffman table. The Huffnaan tables are generated by ranking ^ngUsh plural fwrns. Only if necessary, one cr more of 
die usage frequencies of the substrings in the block lists; 20 eleven lists of "inegular" singular forms, representing 
more frequently used substrings arc given shorter Huffinan exceptions to the general rules for EngUsh singularization, 

are binary searched for the word. If the word is not present 

^ r.. . . • ». . in the irregular lists, it is singularized according to the 

vn. Jhc Singularization Function ^^^^ ^ singularized acccrding to the 

To ensure that text is accurately rqxioduced, the lexical ^ irregular rule. Words not falling into the plural categories are 

parser* s 120 singularization function must be reversible: diat singular ized. 

is, if a word is detcnnined to be plural, and then singularized Table 9, below, describes in pseudocode the slngulariza- 

by the function, there must be a coniq;>lementary pluraliza- tion me&od of the preferred embodiment The eleven Irregu- 

tion function that exactly reproduces the original word. This lar lists were derived from a study of 130,000 of the most 

goal is more difficult than it at first appears, since many common EngUsh words. Words in the SES-to-^IS list, for 

words may be pluraiized in nxire than one way (e.g., zeros example, include analyses and neuroses; these words are 

cr zeroes). If, for example, the w<^ **zen>s" is encountered transformed to analysis and neurosis, rather than analyse and 

and converted to '"zero** by (he lexical parser 120, should it neurose, as the general role would dictate. Words in the 

be re-stenuned to "zeros" <v "^oes**? For most common CHES-to-CHElist include psyches and quiches; these words 

words, the problem is merely semantic and may be over- are transformed to psyche and quiche, rather than psych and 

looked; but since text documents frequently include proper quicb as the general role would dictate. 



TABLE 9 



if length <3 then return, 
if Ust letter ic 

s: if seccod'tD-last letter is: 
e: if dud-to-last letter is: 

s: if in dte SBS•4>S^S list, change -set k> -sis, 
else if fouith-t>Iut ktter is: 
a: if in the ASBS-lo^ list, dnoge -oses to -as, 

else cha ngp -a ses to -aae. 
i: if in the ISB5-tt>-IS list, obsago -ima* to -is, 

else change -tses k> -iae. 
o: if m the OSES-lo-OS list, change -o*ef to -os, 

else change -otes to -ose. 
u: if in the USES-to-US list, and 00c m the 
I-to-US list, change -ims to -us, 
else change -uses to -use. 
s: changp ^ses to -cs. 
otherwise, change •ses to •fe. 
c: if in the CES-to-X list, change -oes to -x, 

else change -oea to -oe. 
x: if faurtb-to-last ktter is a, chaogc -axes to -axe, 
else if not in C£S4o-X list, change -xes to -x, 
else return. 

v: if in the VES-Id-F list, cfaazve -ves to -t 

else change -ves to -ve. 
t: if in the IBS-to>IB list, change -ies to -ie, 

else if fourtb-to-last hetler u oot ■ vowel, 

change -ies to -y, 

else return- 

o: if in the 0£S-to^ list, change -cea to -o, 

else change -oca to -oe. 
h: if fburth-fio-last leSer is: 

s: change -ahes to -ah. 
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TABLE 9^contuiued 



c: if in CHBS-40-CHE list, duogc -ches to <be, 

else '^ttt^g i^ <lies to "Ch. 
odieiwise, cfamge -hes to -he. 
z: if founfa-to-last letter is z, and not in 
the ZES to-Z list, charge -zzes to -z, 
else if in tbe ZES-to-Z last, change -zes to -z, 
else change -zes to -ze. 
otheiwiae, chai^ •es to -e. 
a: if in die ASES-to-AS list, return, 

else change -as to -a. 
i: if in tbe SES-to^IS or ISES-to-IS lists, return, 

else oonveit -is to -I 
o: if not in the &st, change -os to -o, 

else ntum. 

u: if tfaiid«>-last letter is a, change -uas to -ua, 
else return. 

y: if tfaiKMD-last letter is a vowel, change -ys to -y, 

else return, 
s: return. 

^ if in the VES-to-F Hst, renim, 

else cluQge -£i to -f 
h: if tfaflfd-to-last letter is c cr s, return, 

else chaQge -bs to -b. 
X'. return, 
z: return. 

n: if tfainMD-last letter is a, and 
fburtiMD4ast leffer ts m, return, 
else change -os to -n. 

otherwise, delete die final 
h if in I-t>US list and not in USES-lo-US list, 

change •! to -us, 

else retum. 
n: if seooQd*to-last letter is e, 

and dnrd-to-last Idter b m, change -ooen to -man, 

else fcturiL 
otherwise, retun. 



The pluralizatioo method corresponding to the above 
singuiarization method is described below in Table 10. 

TABLE 10 



if bst letter is: 

s: if kogtfa >2,lhen 

if secood-eo-last letter is: 

L- if in die SBS-to^IS list, chinge -is to -es. 
else change -b to -ises. 

u: if in dK USES-to-US list, change -us to -uses, 
else change -m to -i. 

otherwise, append -cs. 

else append -^s. 
o: if in the OES-to-O list, append -«s, 

ebe append -a. 
y: if second-to-last letter b a vowel, append 

else change -ys to -les. 
b: if secoctd-to-bst letter b c or s, append -es, 

else "fT^P*^ 4. 
z: if in die .CBS-to-X list, chai^ -z to -oes; 

else append <a. 
z: if in die ZES-to-Z list, append -ea, 

else append -zes. 
f: if in die VES-to-F list, change -f to -vea, 

else append 4. 
n: if secood-to-last letter b a, 

^T>rf tfaird-4o-last letter b m; -ooao to ^nea, 

else append •«. 
otfaerwbe, append •s. 



Vm The -Ed and -Ing Stemming Functions 

Like the singularizatioo function, the -ed and -ing stem- 
ming functions must be fuUy reversible so that the original 
text is accurately reproduced. Again, these functions could 
be implemented using a dictionary of word forms, but such 
an approach would significantly slow the operation of the 



33 lexical parser* s critical inner loops. For this reason it is 
I»'eferable to use rule-based moq^ological transfonnations 
for these functions. 

For most regular English past tense, past particq)le, and 
present paitidple inflections, the only si^^cant question to 

40 be resolved in forming tbe word's lemma is whether an e 
should be qjpended to the word after the -ed or -ing suflSx 
is removed. In the i^fccred embodiment, this decision is 
made by consulting a three-dimensional binary matrix in 
which the coordinates of each bit in the matrix are the last 

43 three letters of tbe word in question, after the -ed or -ing 
suflBx is removed. 

Each bit in the matrix coirespoods to a particular triple 
word ending, as noted above. If the bit is one. an e should 
be appended to words having that ending; if zero, it should 

^ not The matrix was generated from a study of 130.000 
comnkon English words. The frequencies of words having a 
paiticular triplet ending which required an e to be appended 
were compared with those words having the same triplet 
ending, but which did not require an e. Bits ooiresponding 

55 to those triplet endings more often requiring an c to be 
appended are set to one. For example, more words which 
end in -bit, after the -ed or-ing suffix is removed — such as 
<^ting or inhabiting— do not require an c to be appended 
to form tbe root, than similarly ended words which do. such 

^ as biting. The bit in the matrix corresponding to the -bit 
ending, therefore, is set to zero. 

The -ed stenunlng function also converts words ending in 
-led to -y, if the letter preceding the -icd suffix is not a vowcL 
Neither function is completely accurate. Note, for 

65 example, that in Table 5 the word nothing is stemmed to 
noth. These inaccuracies are. however, infrequent, invisible 
to the user, and mosUy harmless, except for the addition of 
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an otherwise unoecessaiy plu&Ed or plusing metacodc. More each keyword is convcited to its lemma, the keywords that 

impoftaotly, however* the functions are exceptionally fast are actually searched ait ^Inan** and "Voman,** which will 

and fully reversible. retrieve all documents containing **man" ot "men," and 

^Voman" or '•women.'*) 

K. Searching the Archive 5 3^ ^^^^ con9>lex keywwd searches can be pcrfonncd 

Because the archive 30 contains an inverted index of all by specifying proximity connecter, such as '^man NEAR 

of the search tenns in all of its documents, and because each woman " or **man within five words of woman,** or '"man 

document contains an inverted index of the positions of all preceding woman by five words.** A proximity matching 

of its search tenns, the archive 30 is particularly well function first evaluates the Boolean equivalent of the query* 

adapted to fast keyword searches, including keyword prox- treating all proximity <^>erator5 as ANDs. The proximity 

imity searches. This section describes the search mechanism matching function then scans each document pointed to by 

used with the preferred embodiment of the invention. the resulting reference vector 42 individually, to identify the 

FIG. describes the process used for performing a pure fuzzy positions of the keywords wifliin each document, and 

keywa-d-in-document search. A keyword transformation ^cn compares these positions as specified by the relevant 

function (not shown) first processes the keyword 41 using proximity operator. To identify the fuzzy positions for each 

the lexical parser's set of transformation functions 122, to keywwd, the proximity matching function need only 

yield a transformed keyword that is the lowercase lemma of decompress and scan the dictionary reference/position vec- 

Ihc criginaL (U a stopword is encountered at this stage, the table 336 for the document Moreover, Ae table 33b need 

search may be halted and the user infonned that the search ^ decompressed incrementally; that is, only the first n 

cannot be performed for that word.) The kcywonl transfor- ^ elements in the table need be decompressed, if a given 

mation function then block compresses the transformed kcywOTd is the nth term in the table, finally, because it is 

keyword using the leaucal parser's term block conqresscr already known which dictionary (standard 35 or archive 36) 

124, yielding a dword representation of tfie keyword 41. Any * particular keyword belongs to, and because the dictionary 

word complrtion code (i.e., bytes beyond six in die block rcference/jposition vector table 336 is written in two parti- 

con^jrcssed search term) is discarded. ^ ^^^^ acccHding to each dictionary, only (he first n dements 

A keyword search function (not shown) then searches the ^f the partition containing the keyword, where n is the 

standard dictionary 35 for the dword, using the dictionary P**"*^"" *^ )^yvi^d wUhin the partition, need be deoom- 

rmoching function 141. If the dword is found there, the pressed /Uar^^ 

search function checks the standard link file 37 at the 3^ ^^^^y P<"°nned. 

loaition corresponding to ^e dward's position in the stan- x. Extracting a Metafile &om the Archive 
dard dictionary 35 to see if a valid index pointer is there. 

(Search terms in the standard dictionary 35 which are not If a desired document's serial number is known, the 
found in any document in the archive are assigned invalid document's metafile 20 r^HCsentation may be extracted 
index pointers.) If, on the other hand, the dword is not found from the archive 30 dirough the process described in FIG. 
in die standard dictionary 35, the search function searches 10. This process is substantially die reverse of the first part 
the archive dictionary 36 for the dword, again using the of the i»occss described in FIG. 6. 
dictionary matching fiinction 141, and if it Is found there. An extraction function first obtains the location of the 
looks up die corresponding index pointer in the archive link compressed document within the document file 33 by read- 
file 3S. 40 ^8 document file pointer corresponding to the known 

If the search function finds a valid index pointer by ddier serial number from the r^on file 32. Next, the extraction 

means, the search function then invokes the sequential function decompresses the compressed document's dictio- 

differendng compressor 142 to decompress the appropriate nary reference^sltion vector table 336 using the sequential 

document reference vector 42 from the index file 34. The differencing compressor 142. The decompressed dictionary 

decoaqHesseddocumentreference vector 42 comprises a list refereix;e4x>sition vector table 336 comprises a list of dic- 

of the serial numbers of all documents in the archive 30 tionary references, each of which has a position vector, as 

containing the keyword 41. described above. Each decompressed dictionary reference is 

Because the keyword is transformed to its regular lenuna "scd to lode up its corresponding dword fi-om die archive 
before searching, and because only regular lenunas are and standard dictionaries 35, 36 using the dictionary match- 
stored in the archive's dictionaries 35, 36, the keyword 50 ing function 141. (The archive and standard dictionaries 35. 
search will retrieve documents containing all regular noun ^ are not shown in FIG. 10 for clarity.) The extradion 
and verb inflections of die keyword, as well. With very few function stores this dword in die metafile's word vector table 
cxcQ>tiotts, regular ooun and verb inflections mean nearly Likewise, the extraction function stores each dccom- 
the same thing as their lenunas. and for this reason such a pressed position vector with its corresponding dword in the 
result is hi^y desirable. 55 WMd vector table 21. The extraction process is then cora- 

Mort complex searches can be easfly performed using a P^*^*^ (Jcconiprcssing the paragrqA descriptor reference 

keyword query 40. TTic keyword query 40 contains a set of ^ ti« adaptive vmablcrate compre«OT 143, and 

keywords, optionaUy linked by Boolean connectors such as d^compressmg die paragraph descriptor table 33/ using die 

AND, OR. and NOT. The search rcsuU for such a keyword ^^^^ con^jressor 144. 

query 40 is Ae lesult of die pure J^ord-in^doounent 60 xL Expanding a Metafile into a Document 
search described above, f w each Iceywcrd, where the result- 
ing document refaence vectors are AND-cd. OR-cd, or To complete the retrieval process, an expansion function 
NOT-cd together, as necessary. For instance, to search for 70 transfonns the metafile 20 into the original document 10. 
documents containing "men AND women,** the reference An expansion function first decompresses the dwords in 
vectors retrieved for 'Inen*' and 'Vomen** are AND-ed 63 the metafile's word vector table 21, using die dword block 
together to yield a reference vector listing all documents con^nessor 124. FIG. 11 illustrates the decompressed word 
containing both "men** and **women.** (Note ttiat because vector table 72 using the sample text of IWe 1. Next, die 
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expansion function maps the dcconqjrcssed word vector linc-deUmitcd text, or the format of a word process), one 

tabic 72 into a pointer list 71. The pointer list 71 is a list of skilled in the ait will appreciate that the expansion fiinction 

pointers to the search terms contained in the decompressed will output the reconstituted text of the document 10 in 

word vector table 72. (In FIG, 11, the actual words arc different fashions depending on the format desired. If, for 

shown in die list 71, but only for clarity. In reality, these 3 exanq>le, ASCH line-delimited text is desired, the expansion 

entries are pointers to the search terms contained in the function will provide end-of-Une delimiters when text 

dcoHnpressed word vector table 72.) reaches the right margin of the "page." If, on the other hand. 

The expansion function maps the decompressed word wwd processor output is desired, ttie expansion function 
vector table 72 into the pointer list 71 by first creating a list will provide only cnd-of-paragraph delimiters at the ends of 
of n null (invaUd) pwntcrs, where n is the largest fuzzy ,q paragra|As, and will preface new paragraphs with the appro- 
position (that is, die fuzzy position of die last word in die priatc margin, leading, and tab set codes required by die 
document 10) stored in die word vector table 21. (The nuU wwd proccssw. likewise, die c:xpansion function wiU trans- 
entries in die example list shown in FIG. 11 are represented late word en^ihasis codes (e.g. , underline or boldface) into 
by cnq)ty spaces.) (As noted above, die fuzzy position of die die apfffopriate emphasis codes required by the word 
last word in die document 10 is stored as part of die processor, if diat type of output is desired, 
document header 33a, so there is no need to scan die word xtt Glossary 
vector table 21 to determine what n is.) Next, the expansion ^ , . , ^ * « r a*. 
function travcxses die decompressed word vector toble 72, ^ glos««y below presents a summary of some of die 
replacing each null pointer corresponding to a fuzzy position tcnns used m dus specification. , ^ , ^ 
at which each search tenn is located widi a pointer to diat Block compression— a concession techmque diat replaces 
search term- In odio- words, if a search term in die deomi- 20 substrings of an input string widi one or more fixed-lcngdi 
pressed word vector table 72 occurs at fuzzy positions 10, codes diat reference a statisticaUy generated table. The 
105. and 2386, die e;q»ansion function replaces die null referenced entry in the table contains die replaced substring, 
pointers at positions 10, 105, and 2386 widiiD die list 71 widi Dw<»'d~a block conQ|Tessedi possibly truncated search 
pointers to that search tenn. tena Dwords are stared in die m^afile word vector table. 

As can be appreciated by one with skill in the art, die 25 and the standard and ^>ecific dictionaries. In the preferred 

combination of the pointer list 71, code list 22, residuals list embodiment, dwords can contain no more than six bytes. 

25, and paragr^ desoiptor reference list 24 together Fuzzy position — a number assigned to a search term in a 

concise a linear representation of die document 10, sepa- document to identify its approximate position in the docu- 

rated into four con^nent parts. The expansion function ment 

dicrefore need only traverse these four lists, using die 30 Inverted structure — an alternative storage structure for a list 

pointer list 71 as a primary guide as to whedier the next of items, in which each item is listed uniquely, together widi 

element in die document 10 is a search term or a metacode its posftiott(s) widiin die list For example, die inverted 

string. If the current pointer in die pointer list 71 is null, dien structure corresponding to the list '*a, b, c, a, b, a, c, a" would 

die expansion function decodes die next metacode string be "»=1. 4,6,8; b=2,5; o=3,7." 

firom the code list 22, by first decon^sressing it using the 35 Keyword search — an information retrieval technique in 

Huffman compressor 126 (which yields a sequence of block which one or more documents are searched to determine 

codes), and then decompressing it using the metacode block whedier they contain a particular keyword. See also prox- 

cotiqnessor 125. The deconq;}resscd metacode string is then imity search. 

used (1) to set ten^Krary word transformation flags, which Keyword query — a common method for conducting a key- 
cause die next search term diat is output to be capitalize4 or 40 word or proximity search, in which one or more keywords 
pluralized, and so on (if, for cxanqAc, the metacode string are specified, optionally linked by connecters such as AND, 
contains die c^ or plural mctacodes); (2) to set ten^x^ary OR, or WTTHIN. 

output format flags, which change the format in which Leading (pronounced 'ledding**) — ^vertical space between 

succeeding text is output (if the metacode string contains die lines of text Primary leading is the vertical space t>etwecn 

format on/off metacode; as can be ^^iredated by one skilled 43 successive lines of text in a paragraph. Secondary leading is 

in the art, the expansion function reads the type of forroat- the vertical space between successive paragrq>h5. 

ting toggled firom die residuals list 25); (3) to signal die Lenuna— the base, uninflccted form of a word. In diis 

beginning of a new paragraph (if the metacode string specification, lenuna may also indicate a partially inflected 

contains the new paragr^)h metacode; as can be appreciated fomu such as the singular form of a present participle. For 

by one skilled in die art, the expansion function determines so exan^le, ranting is die lemma (albeit the singular lemma) of 

die format of the new paragri^h by reading the next refer- rantings; rant is the completely uninflected form, 

ence from the paragraph descriptor reference list 24, which Line-delimited document — a document in which each line 

identifies the paragn^ descr^>tQr code in the paragrt^h of text is ended (delimited) by a special symbol usually a 

descrq>tor taWe 23 which describes the new paragr^); (4) ^'return" character. Compare with paragraph-delimited docu- 

to output stopwords (first transformed, of course, by any of $5 ment 

the current word transformation flags, which may have been Metacode — a code that describes punctuation, formatting. 

set by preceding metacodes); and (S) to output punctuation. stopwords, and/<v morphological transformation flags that 

If, on the other hand, the current pointer in die pointer list 71 occur between search terms in a document. 

is valid, the expansion function outputs the search term Metacode string — a sequence of metacodes. A single meta- 

pointed to by that pointer, first transformed as required by 60 code string represents all of the punctuation, fmmatting. ete.. 

any of die tempcHary word transformation flags (which were that occurs between two search terms. 

setby the preceding metacode string). If two adjacent search Metafile— an interim storage structure that represents the 

terms or stopwc^ds are encountered without any iotervemng original document in partially inverted form. 

punctuation, then the expansion function outputs a space Morphological transformation — the transformation of a 

character to separate them. 65 word into another form, as in changing opened into open. 

Because the user may desire to reproduce the original Paragraph-delimited document — a document in which each 

document 10 in any of a number of formats (e.g.. ASCII paragraph of text is ended (delimited) by a special symbol. 
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usually a **retum** diaracter. The lines of text in each 
paragraph are not delimited, and thoefore must be 'l^oken** 
or "Vcffd wrapped'* at the margins^ in order to be displayed 
or printed. Compare with line-delimited document. 
Paragraph descriptor code-^ data structure that defines the 
format of a single paragraph. In the preferred embodiment of 
this invention, a paragraph descriptor code indicates the left 
and right boundaries (margins) of the paragraph; its fint line 
indent or outdent amount; its primary and secondary lead- 
ing; its justificaCioo; and whether its text is in all uppocase 
letters. 

Pica — a unit of measurement; one pica equals 0.1 inches, the 
width of a single character of a mooospace font having ten 
characters per inch. 

Pointer — a number that indicates the absolute byte position 
of an element in a list, usually a list of elements having 
nonunifonn sizes. Compare with reference. 
Etimary leading — the vertical spact t>etween successive 
lines in a paragraph; see leading. 

Hx>ximity search — a type <rf keyword query in which two or 
more keywords are linked by a proximity operator such as 
**near- ca* *\(ritfain n words or ot •"precedes by n wonls." For 
example, *%ian widiin/5 woman" indicates a request for all 
documents having die word man located within five words 
of the word woman. 

Reference — a number that indicates either the absolute or 
relative position of an elenient in a list of uniformly sized 
elements. The absolute byte position of the element (a 
pointer) can be calculated from the reference. 
Residuals — inf<Hmation about a document that cannot be 
easily or efficiently stcned in the metafile's word vector 
table, code list, or paragraph f<moai structures. 
Search tenx> — any word (consisting of only alphanumeric 
characters or, possibly, embedded periods or apostrophes in 
the preferred embodiment) that is not a stopword. 
Secondary leading— the antount of space between succes- 
sive paragr^hs; sec leading. Secondary leading can be 
viewed as the vertical space before a paragraph, after it, or 
both; in the preferred embodiment, secondary leading is 
treated as the space befrn the paragr^. 
Sequential differencing — a technique for reducing the aver- 
age magnitude of a list of ascending numbers, where each 
number is replaced by the difference between it and die 
number before it For exaoGple, the list "1, 4, 8, 17. 18. 20" 
would be transformed to "1, 3, 4, 9, 1, 2" by sequential 
differencing. 

Stopword — a wc^ not treated as searchable because it has 
low information content (often called **function words**). 
Examples of some typical stopw<sds are the. of, and for. 
Table— a vector of items, usually items containing vectors. 
Vector — a one-dimensional list of items, usually references 
or pointers. 

Word completion code — that pcxtion of a block compressed 
search term that cannot fit wittiin a dwcsd. Word completion 
codes are. in the preferred embodiment stored in the residu- 
als list 25. 

This invention may be embodied in other ^)ecific fonns 
without departing &om its spirit or essential characteristics. 
The described embodiments arc to be considered in all 
respects only as illustrative and not restrictive. The scope of 
the invention is therefore indicated by the appended claims 
rather than by the fofcgoing description. All changes which 
come within the meaning and range of equivalency of the 
claims are to be embraced within that scc^. 

I claim: 

1. A computer-based system for storing text, comj^ing: 
a lexical parser which divides said text into search terms 
and all otha information; 



a word vector table which stores said search terms solely 
as an inverted structure, wherein each said search term 
is stored uniquely and each said search term is coupled 
widi a list of its positions within said text; and 
^ a first auxiliary data structure which siores only said other 
information; 

wherein said word vector table and said first auxiliary data 
structure contain sufficient information substantially to 
reproduce said text 

10 2. The system of claim 1. wherein said positions are 
cotmted from the beginning of the text, each search term In 
said text is counted as one position, and each instance of 
other information between said search terms is counted as 
one or mo'e positions. 

1^ 3. The system of claim 1. wherein said text contains 
words, and said search terms are the sppronlmatc lemmas of 
said words. 

4. Hie system of claim I, wherein said text contains 
words, and said search terms are compressed representations 

20 of said words. 

5. The system claim 1, wherein said lists of positions 
are conqHessed before storage in said word vector table. 

6. The system of daim 5, wherein said lists are com- 
(H-essed at least in part by sequential differencing. 

^ 7. The system of claim 1, wherein said search terms are 
stored in said word vector table by reference to a second 
auxiliary data structure, said second auxiliary dau structure 
storing said search terms. 

8. The system of claim 7, wherein said text contains 
^ words, and said search terms are the i^^ximate lemmas of 

said words. 

9. The system of daim 7. wherein said text contains 
words, and said search terms are conqnessed representations 
of said words. 

35 10. The system of claim 7. wherein said lists of positions 
are compressed before storage in said word vector table. 

11. The system of daim 10. wherein said lists of positions 
are compressed at least in part by sequential differencing. 

12. The system of claim 1, wherein said text conqnises 
^ one or more paragraphs having paragraph formats, said 

formats including at least the mar]gins o( said paragraphs, 
and wherein said paragraph formats are stored in said first 
auxiliary data structure as a table of said formats and a list 
of references to said table. 
^5 13, A computer-based method for storing text conqxising 
the steps of: 

dividing said text into search tenns and all other infor- 
mation; 

^ storing said search terms solely as an inverted structure, 
wherein each said search term is stored uniquely and 
each said search term is coupled with a list of its 
positions within said text; and 
storing only said other information in a first auxiliary data 

3j structure. 

14. The method of daim 13, wherein said text contains 
wcvds, aiKl said search terms are the ^)proximate lemmas of 
said words. 

15. The method of daim 13, additionally comprising the 
^ step of com^essing said inverted structure after said first 

storing step. 

16. A con^MJter-based method for compressing a particu- 
lar string of symbols, con^xising the steps of: 

providing a large number of strings similar to said par- 
65 ticular string; 

assigning a value to nearly all substrings contained within 
said large number of strings, said value being equal to 
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the frequency of occurrence of said substring in said 
large number of strings, multiplied by the length of said 
substring; 

storing the highest valued sut>strings in a list; 

assigning to each substring in said list a unique fixed- 
length code; and 

replacing each substring contained both within said j>aF- 
ticular sthng and said list with the code corresponding 
to said substring. 

17. A computer-based method for compressing a particu- 
lar string of symbols, conqsrising the steps of: 

providing a large number of strings sinoilar to said par- 
ticular string; 

providing a temporary list of substrings contained within 
said large number of strings; 

assigning a value to ail substrings in said teII^>Qrary list, 
said value being equal to the frequency of occurrence 
of said substring in said large number of strings, 
multiplied by the length of said substring; 

subtracting from each string in said large number of 
strings each substring contained in said temporary list; 

assigning a value to all remaining substrings in said large 
number of strings « said value being equal to the fre- 
quency of occurrence of said substring in said large 
number of strings* multiplied by the length of said 
substring; 

replacing all substrings in said temporary list having a 
value lower than any said remaining substring with the 
highest valued said remaining substrings; 

iterating said assigning, subtracting, assigning, and 
replacing steps until said temporary list is substantially 
unchanged from the previous iteration; 

assigning to each substring in said temporary list a unique 
fixed-length code; and 

replacing each substring contained both within said par- 
ticular string and said list with the code corresponding 
to said substring. 

18. A computer-based method for storing text, comprising 
the steps of: 
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dividing said text into search terms and all other infor- 
mation; 

compressing said seardi terms using the method of claim 
16; 

storing said search terms solely as an inverted structure, 
wherein each said search term is stored uniquely and 
each said seardi term is coupled with a list of its 
positions within said text; and 

storing said other information in a first auxiliary data 
structure. 

19. A computer-based method for storing text, comprising 
the steps of: 

dividing said text into search terms and all other infor- 
mation; 

storing said other information in a first auxiliary data 
structure; 

compressing said search terms using the method of claim 

storing said search terms in a second auxiliary data 
structure; and 

storing said search terms in an inverted structure, wherein 
each said search term is stored uniquely by reference to 
said second auxiliary data structure, and each said 
search term is coupled with a list of its positions within 
said text 

20. Aconqxiter-based method for storing text comprising 
the stq)5 of: 

dividing said text into seardi terms and all other infor- 
noation; 

storing said search terms solely as an inverted structure, 
wherein each said search term is stc»ed uniquely and 
each said search term is coupled with a list of its 
positions within said text; 
conq^ssing at least part of said other information using 

the method of daim 16; and 
storing said other information in a first auxiliary data 
structure. 
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